Introduction to Secrets Rotation

Mateusz Wilczyński CTO

Rotation is the process of periodically updating a secret. The secret can be an API key, security credential, encryption key, or any other sensitive value.

Because secrets are sensitive to leakage or exposure, it’s important to rotate them often, at least every 60 days.   Microsoft Azure Documentation
The manual rotation process can be time consuming and human error prone, as a result making the process neglected. In this article, I will present how to automate the rotation process using the AWS Lambda service and the tradeoffs of this method. 

Why rotate secrets

According to community best practices, secrets should be rotated regularly, let’s look at the reasons why.  Minimalizing the consequences of secrets leak is one of the main reasons behind secrets rotation. In a secrets leak event, a bad actor gains an access to one or more of your secrets. Such action often remains undetected for an unacceptably long period of time. This gives the attacker a chance to investigate your system and spread the attack over time, which decreases the chances of detecting the intrusion manually or by anomaly detection systems.  Secrets can leak in various ways, but disturbingly often this happens by putting them as plain text in Git repositories. According to the GitGuardian report, over 6M secrets were detected in public GitHub repositories in 2021.  Inaccurate employee offboarding is another process that can result in unauthorized persons having access to secrets. A former employee having a copy of still used API keys can easily become a security risk.  As you can see, automatic secrets rotation will not solve the described issues, but it can decrease the bad consequences of compromising secrets even if you are not aware that the secret has been compromised.  An important feature of secrets rotation is the possibility to rotate secrets on-demand. You can use it in addition to the automatic rotation every time when there is a suspicion of a secret being compromised. 

Where to store secrets

This is a wide topic, but as you probably know, secrets shouldn’t be hardcoded directly in the application code. Generally, in the AWS cloud, we recommend storing secrets in the AWS Secrets Manager. If you’re looking for a cloud agnostic alternative, we can honestly recommend Hashicorp Vault. 
If you are using Terraform, check out our module that simplifies the secrets management for ECS containers. Get it now
AWS Secrets Manager has built-in integration for rotating Amazon Relational Database Service (Amazon RDS) credentials. It also uses secrets versioning – such a mechanism allows uninterrupted application execution when secrets are rotated.

How to rotate custom secrets

Custom secrets, like external API keys, stored in Secrets Manager can be rotated using a bespoke AWS Lambda function.  In short, this process consists of creating a new secret, saving it in Secrets Manager, switching the application to use the new secret, and expiring the old one. 

Master Secret

In the scenario where the AWS Lambda, or any other custom code, is responsible for generating a new API key for a given service, such Lambda has very high permissions. To reduce the risk, it’s a good practice to have a dedicated Master Secret for a given API that allows to generate new secrets and use it only in the Lambda responsible for rotation.  Such an approach requires granular permissions for API keys to be implemented on the API provider side. There needs to be a possibility to create a Master API key, allowing the creation of standard API keys that can be used by the application but don’t allow creating a new API key.  Keep in mind that compromising the Master API key can also happen, it’s good to be prepared for it. 

Conclusion

Now you understand why rotating secrets is a good practice. It can mitigate risks associated with secret leaks even if you are not aware that a given secret has been compromised. Keep in mind that it’s not a silver bullet solution. If the bad actor would use compromised secrets immediately, he can perform the attack before secrets are rotated automatically. Properly handling the Master Key is another challenge of this technique.  Taking into consideration the downsides, secrets rotation is still beneficial and should be a required element of your security strategy.