Secret Leaks Handling
You found a secret leak and are gasping for breath. Perhaps you committed accidentally an .env file with development or production credentials to a public repository…
Don’t panic ! Promptly following the adequate steps avoids most of the potential damage that bad hands could do with the leaked secret. Balance the assume total compromise stance with the do the right thing first approach. Ths guide gives hints on the steps to follow for dealing with the problem before it is too late.
To summarize, the following cheatsheet illustrates the procedure that will be discussed in what follows.
Important Terms
Definitions of some important terms follow:
Secret
In this guide the term secret refers to any data used in authentication or access control, and other security services. Common secrets are access tokens, API keys, credentials like username / password, cryptographic keys, etc.
More generic sensitive information leaks have a much broader scope. We will restrict here to the types of secrets listed before.
Secret Vault
Also known as Secret Management system, is a centralized secure storage and management solution designed to protect sensitive data such as passwords, API keys, cryptographic keys, and other confidential information. There is a wide range of commercial on-premises, cloud-hosted and open-source products in this space.
Secret vaults provide capabilities for rotation and versioning, which are essential for handling secret disclosures. Secrets are encrypted in storage and transit, and can be fetched in clear-text by consumers after strict access control. All accesses are logged and monitored for misuse.
Exposure Window
The period of time between when the secret is leaked and when it is detected and fixed/mitigated. This vulnerability window could be short if an automated secrets scanner detected it quickly, or long if it was reported by a third party, or even worse if discovered after a breach using the secret occurs.
Immediate revocation of the secret is not always possible. And when the leak happens in a publicly accessible location, such as a public git repository, even a short exposure window is bad news. Threat actors have bot networks that routinely scan public code and container image repositories for hard-coded secrets.
Basic principles
Deleting a hard-coded secret in a git repository does not work
You can delete the hard-coded secret in the file and publish a commit with the change. You can also make the repository private, or even delete it. No way, anyone who cloned and pulled the repository and got the commit with the leak will have access to the secret and do his/her magic !
Git keeps full commit history, and even if you rebase history, remote copies are not updated automatically, you cannot force that. The bad actor simply needs to copy the secret. Once the secret is leaked, it is better to assume that it was compromised. This leads to the next principle:
The same goes for other registries, such as package, IDE extension, or container image registries: once published, deleting the version with the leak, if possible, does not guarantee that the secret has not been captured.
The bad guys often search for secrets in public repositories, and have bots doing this work 24x7. You can even test this with canary tokens, that will notify you when a fake secret placed where "nobody should be poking around" is accessed.
Assume total compromise
Putting you in the worst case is the best way to prepare for the unexpected. Always remember that credential exposure is the underlying cause of many devastating cyberattacks. Even when in private repositories (could you trust third-party developers with access to the repository?) or for internal systems (network perimeter is less clear with cloud and remote work).
What does "assume total compromise" mean in this context? Giving for granted that the leaked secret actually fell into bad hands, who immediately used the secret in the most damaging way possible. That means that you must have a published plan for rapid response, perhaps taking ideas from this guide.
Different remediation approaches
Depending on the nature of the secret, there are different ways to fix the issue:
Revocation, where the access token is eliminated and cannot be used for access. This is common for access tokens and API Keys. Other terms, like expiration, are nearly synonyms.
Key rotation or renewal, where the key is changed and the leaked old key is not valid anymore. The target systems using the key need to get the new key updated. A password reset follows this approach. Sort of revocation plus generation of a new key. This is common with cryptographic keys and certificates. If the key in question is already under regular key rotation, the process probably is well-oiled.
Account suspension. The secret is not changed, but the owning account is disabled / set inactive. The secret (often a credential) could be renewed and then the account reactivated.
Blacklisting, where the secret is put in a black list that the target system uses to reject accesses using it. The secret remains valid and could be removed from the blacklist, but after a leak this is quite dangerous.
Some types of keys, such as private/public key pairs or SSH keys, come in pairs. There is a private key (the secret) and a corresponding public key (or derived public key certificate), and both need to be reissued and changed at the leaked key pair locations.
Order matters
There are different steps during the handling of a secret leak: impact assessment, checking for unintended usages of the secret, including anomalous accesses and operations in the affected systems, performing the revocation / rotation / suspension, removing the secret from the sources where it was disclosed, or making changes to prevent future leaks or reduce the impact of future leaks.
But the order is important, to limit the exposure window between the leak and the remediation. Probably revoke-first is appropriate for most secret-related issues.
Location does not matter
The secret leak could appear in different points: a logfile, a word document or excel sheet, a variable in source code or script, a comment in source code, a binary file, a 'tarball' for a container image, or a ticket. It does not matter which container is exposed: a secret leak has occurred.
Secret leaks in container images could appear in any layer, even though the secret does not appear in the merged container file system.
Response Procedure for Secret Exposure
This is obviously specific to the system that owns the secret ("secret host"), and to the systems that uses it for authentication, access control or cryptographic operations ("secret users").
First things first. Follow your organization’s standards for handling cybersecurity incidents. Hera are the common steps to follow. Change the order if necessary: immediate containment by revoking or rotating the exposed secret is often the first thing to do for some critical systems.
1. Quick Assessment
When you know about the leak, do a quick assessment. Follow this checklist:
understand what the secret provides access to: what services and data a potential attacker can access with the secret?
know what could break after secret rotation / revocation, or account inactivation. The first line of defense can affect systems availability, as the new secret needs to be installed in the systems that rely on it. Having centralized secret vaults make this much easier, as there is a single point for updates. Having also an inventory of which secrets are consumed by which applications / services reduces the risk.
quick risk assessment. How critical are the data or systems affected? What is the exposure window?
TIP; Do not search now in the logs if the secret has been used, unless you have tools in place for it. This is typically a post-rotation analysis. Ask for help to your DevOpts or IT Operations team. Even if there are no questionable accesses, it is still important to move ahead with the secret rotation.
2. Notify the parties
All affected parties need to be aware of the issue, and the potential consequences.
The parties to notify could include:
CISO and internal security team.
DevOps / IT Operations team.
Project manager.
Other managers and executives.
In particular, if you are aware that the secret was used before been revoked, other parties would need to be notified:
External incident response provider.
Law enforcement, if the breach needs further investigation to identify and prosecute malicious actors.
Legal counsel. Consult with your legal team the implications of the breach, and ensure compliance with relevant laws and regulations.
Affected partners, vendors and customers. We are all in a chain !
Regulatory bodies, like data protection authorities, to comply with legal requirements.
Affected individuals, so they can take steps to protect themselves.
Explain in simple terms which systems were potentially affected, what could happen if the secret is abused, and what is the ongoing remediation process.
A cybersecurity incident response plan typically documents who should be notified and how. Follow the procedure.
3. Rotate the Secret
Depending on the nature of the secret, "rotation" is one of the remediation approaches listed before.
This is obviously specific to the system that owns the secret ("secret host"), and to the systems that uses it for authentication, access control or cryptographic operations ("secret consumers"). Some systems offer an api call or a tool command, some require interaction in a web interface, because administrative rights are often needed and strong user authentication is a must.
First, locate the secret host and find the procedure for:
Create a new secret for the target host.
Replace the leaked secret with the renewed one into the secret consumers.
Revoke the leaked secret. Remember that revocation of a secret could affect its consumers.
Typically, you first create a new secret, replace the old with the new in all secret consumers, and only then revoke / invalidate the leaked secret. Alternatively, immediate revocation could go first, then creating the new one, and installing the renewed secret into all the secret consumers.
Creating and revoking secrets typically require administrator rights - ask for help to your DevOps or IT Operations.
Replacing the leaked secret with the new one is a simple task if secret consumers get the secret from a secret vault. This could be done in a single location, streamlining the process.
Otherwise, an inventory of all the application and services (such as build & deploy pipelines) that use the secret might be useful.
Ask the right team in your organization (DevOps engineering, software development, IT operations…) to perform the exchange.
4. Fix broken things
A credential secret is often used in software build and deploy pipelines. Check that the builds work with the new secret. The Software Development / DevOps teams help with that. If the secret affects applications in production environments, due care should be taken.
The team in charge could test (perhaps using a staging environment) that the affected systems work successfully with the new secret.
Keep in mind that it is not possible to revoke the leaked secret until all the secret consumers have the leaked secret replaced.
5. Investigate in Detail
Once the issue is contained and the leaked secret cannot be abused, it is time for a more thoughtful investigation. Once the secret host is known, access logs could be reviewed for unauthorized operations using the secret.
Ask for help from your DevOps / IT Management teams, as this is specific of the target system. Please note that even with no evidence of unauthorized accesses, secret rotation should be done in any case.
You may use tools that could check if a given secret is disclosed in public repositories or even in the 'Dark Web'.
If unauthorized operations are detected, a more complete review of potential related systems should be performed. The leaked secret could be used to open the doors to other systems. The exfiltrated data could be personal, health-related, or financial-related. Other credentials could be extracted by the attacker. In certain cases, you may need to call an Incident Response firm, and perhaps notify an update of the incident to law enforcement or a regulatory agent.
6. (Optional) Delete the hard-coded secret
Remember that deleting the hard-coded secret does not prevent a avoid a malicious actor to misuse it. But some organizations need to do such deletion, for different reasons:
Reputation management.
Compliance and auditing. Many organizations have compliance requirements that mandate the removal of sensitive information from public repositories. Removing hard-coded secrets helps demonstrate that the organization is taking necessary steps to comply with these regulations.
If you do not delete the hard-coded secret, a secret scanner may flag again the leak for an already revoked secret, in particular if the scanner cannot perform an automated verification to see if the secret remains active. You can use the scanner’s "mute issue" options to forcibly remove it.
You may simply remove the hardcoded secret in source code or comment, or a configuration file accidentally committed to version control, or even modify the git history to remove any trace of the leak, at least in the official code repository.
Rewriting git history is complex and can wreak havoc if not done properly.
For removing secrets in git history, there are good tools like git filter-repo or BFG Repo-Cleaner, that are useful to avoid cumbersome git rebase commands. You may follow the procedure listed here for GitHub.
7. Avoid a similar secret to be leaked in the future
Now lesson’s learned: Review the incident and why it happened. How it was detected and handled, and were you can improve.
If the secret was leaked accidentally during a commit to a git repository, a good idea is to set a secrets detector in a pre-commit git hook. This blocks any commit that inadvertently discloses a secret.
The most important thing is to prevent that the secret is leaked again. You should define an alternate mechanism for getting the secret at the places where it is used, like using environment variables or local files not under version control. With local files, you may need to enforce that the exclude patterns in .gitignore or .dockerignore configurations are properly excluding the secret-holding files.
A good alternative to environment variables and configuration files is to have a central secret vault:
Cloud providers typically offer their own secret management services.
Independent commercial and open source secret vaults are available, such as HashiCorp Vault or CyberArk Conjur.
When the secret is written during code generation in an automated script, you should modify the hard-coding of the secret with logic that grabs it from a safe source.
Last updated
