Secrets Scanner

• Purpose

• Main Functionalities

• Secrets scanner execution

• Xygeni secrets operation modes

• Results upload

• Operations (--scan, --diff, --history)

• Searching secrets under Git (--history, --all-files, --staged-files)

• Preventing introduction of secrets (git hooks)

• Secrets Verification

• Secrets Remediation

• Secrets Auto-Remediation

Purpose

A secret refers to any confidential and sensitive information that must remain secure. This encompasses passwords, API keys, cryptographic keys, tokens, and other credentials used for authentication or authorization. Additionally, secrets may include various forms of sensitive data, such as financial or medical records, personally identifiable information (PII), or intellectual property.

Leaked secrets in version-controlled software projects pose serious security risks to its owners and to projects depending on it.

Secrets gained by bad actors are the cause of major attacks to the software supply chain. Having hardcoded secrets in software source code or configurations, particularly when under source control, is an invitation for bad actors to gain access to organization’s sensitive resources.

Accidentally hard-coding a secret (imagine a JWT token, a production database password, or AWS credential) is easier than you may think. During development the secret is used for testing, but the secret is not removed and a bad .gitconfig makes the secret entering git: One git push, and it is too late.

If important secrets like credentials, API keys, access tokens or cryptographic keys are exposed, bad actors can use them to extract sensitive information, insert malware, move laterally into the victim organization, or even modify source code, binaries or DevOps configurations and scripts to launch supply-chain attacks. Attackers may use your cloud resources to run crypto miners and other nasty things. In short, a single commit with an unintended secret throws the key for the bad guys to come in.

Xygeni Secrets detects hardcoded secrets. Xygeni Secrets performs thorough scans of code, text files and docker images to identify exposed secrets (API keys, passwords, and other sensitive credentials). Such exposures can be exploited by malicious actors to leak data or gain unauthorized access to critical systems.

Xygeni Secrets allows you to determine:

• What secrets have been committed to your repository.

• Scan your repository's history thoroughly for hidden secrets. Enable continuous historical scanning to detect and identify valid secrets throughout your Git history effectively. Protect your repository from vulnerabilities by maintaining robust secret detection practices.

• The validation status of the secret; for example, valid secrets are those that have been tested against a service and confirmed to successfully grant resources or authentication.

Xygeni Secrets saves security engineers time and effort by prioritizing valid leaked secrets and informs developers of valid secrets in their PRs and MRs by posting comments directly.

Once Xygeni Secrets detects a hardcoded secret, it obfuscates the secret before logging it or sending it to the Xygeni platform, ensuring the secret isn't exposed in its output.

Main Functionalities

• Detection of Committed Secrets: Identify secrets already in your repositories, at both repository- and organization-level.

• Pre-Commit Secret Blocking: Prevent secrets from being added to your repositories.

• Comprehensive Git History Scanning: Examine your repository's entire git history for hidden secrets.

• Secret Validation: Confirm the validation status of secrets to ensure they grant access or authentication.

• Docker images scanning: Identify secrets in your docker images

Secrets scanner execution

Secrets scanner can be executed in two different ways:

1.- By its own specific command ( xygeni secrets [options] )

2.- By the general command ( xygeni scan --run="secrets" [options] )

xygeni secrets admites more configuration parameters than xygeni scan.

Although most of the functionalities of secrets scanning are available in either of those modes, advanced functionalities are only available by using the xygeni secrets

Xygeni secrets operation modes

Xygeni secrets has different operation modes depending on the scope of the scan.

• scan : a single scan unit (it will correspond to a single Xygeni project)

• multi-scan : multiple scan units (it will correspond to multiple Xygeni projects)

• org-scan : a whole scm organization's repos (it will correspond to multiple Xygeni projects)

• docker image scan : to scan a docker image

See Xygeni CLI operation modes for furhter information.

Results upload

An important difference is that when using xygeni secrets the results will not be uploaded by default to Xygeni servers. Instead, using xygeni scan --run="secrets" the result will be uploaded by default.

If you want xygeni secrets to upload the results, use xygeni secrets --upload

If you don't want xygeni scan to upload the results, use xygeni scan --no-upload

Operations (--scan, --diff, --history)

xygeni secrets has three different modes of operation:

Operations available:
      --scan                 Scan for secrets (default operation).
      --diff                 Scan for secrets, reporting diffs from baseline.
      --history              Scan for secrets in git history.

--scan

The default operation mode is --scan (i.e. scan for secrets), you don't need to explicitly specify it.

By executing xygeni secrets , the scanner creates a "baseline", i.e. a file (.secrets.baseline.json) with the results of the secrets scanning.

--diff

Let's imagine now that, after running the scanner over a directory, you modify the contents of the source directory and want to know the differences between them. If this is your case, you may run xygeni secrets --diff to know it, i.e. this command will show to you the differences between them.

For example, imagine that we run xygeni secrets over a source directory. Such command will find the secrets and saves the baseline.

2024-07-31 08:59:08 INFO  Xygeni-Scanner - Secrets baseline saved to file: C:\LGV\src\Secrets\secrets_local\.xygeni.secrets.baseline.json

Secret leaks found: 9
┌────────┬────────────┬────────────────┬───────────────┬────────────┬────────────────┬────┬────────┐
│Severity│    Type    │Secret          │Key            │  Detector  │File            │Line│Tags    │
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│critical│github_token│ghp_PPAuLz***...│               │github_token│org_scan.bat    │4   │verified│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│critical│gitlab_token│glpat-DNE_***...│valid1         │gitlab_token│src/AddPage.java│41  │verified│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  high  │gitlab_token│glpat-sPSs***...│invalid        │gitlab_token│src/AddPage.java│48  │inactive│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  high  │gitlab_token│glpat-EAs2***...│valid3_torevoke│gitlab_token│src/AddPage.java│51  │inactive│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  high  │    jwt     │xya_eyJhbG***...│valid3         │xygeni_token│src/test.py     │6   │verified│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  low   │    jwt     │eyJzdWIiOi***...│invalido       │    jwt     │src/test.py     │4   │inactive│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  low   │    jwt     │eyJzdWIiOi***...│valid3         │    jwt     │src/test.py     │6   │inactive│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  low   │    jwt     │xya_eyJhbG***...│valid4_toexpire│xygeni_token│src/test.py     │7   │inactive│
├────────┼────────────┼────────────────┼───────────────┼────────────┼────────────────┼────┼────────┤
│  low   │    jwt     │eyJzdWIiOi***...│valid4_toexpire│    jwt     │src/test.py     │7   │inactive│
└────────┴────────────┴────────────────┴───────────────┴────────────┴────────────────┴────┴────────┘

Next, we modify the source code introducing a new secret. If we execute xygeni secrets --diff , the scanner will show up the differences with the baseline (i.e. one new secret introduced)

Secret leaks found: 1
┌────────┬────────────┬────────────────┬───────┬────────────┬────────────────┬────┬────────┐
│Severity│    Type    │Secret          │Key    │  Detector  │File            │Line│Tags    │
├────────┼────────────┼────────────────┼───────┼────────────┼────────────────┼────┼────────┤
│critical│gitlab_token│glpat-PFdC***...│comment│gitlab_token│src/AddPage.java│45  │new     │
│        │            │                │       │            │                │    │verified│
└────────┴────────────┴────────────────┴───────┴────────────┴────────────────┴────┴────────┘

--history

Finally, the third operation mode is --history. By using history mode of operation, secrets scanner will search for secrets in the whole git history. This option allows you to discover secrets that are hidden into the git commit history and, being valid, could be discovered by anyone with access to the repository. Those secrets found in git history as tagged as history so you can easily differentiate them.

See Searching secrets under Git for further info.

Searching secrets under Git (--history, --all-files, --staged-files)

There are certain parameters related to git that you should take in mind.

Below command only works with Git version control systems.

--history

By using history mode of operation, secrets scanner will search for secrets in the whole git history. This option allows you to discover secrets that are hidden into the git commit history and, being valid, could be discovered by anyone with access to the repository. Those secrets found in git history as tagged as history so you can easily differentiate them.

To detect secrets leaked in any branch, for each commit in the project history, you may use the --history operation. This will analyze all the changes, for identifying when the secret was leaked for investigation of an incident related to a secret leakage.

xygeni secrets --history -n "TEST"

As the history log could be huge for large projects, you may filter by branch or by commit range, if you know approximately where the secret could be located.

--all-files

By default, if the source directory is under git, secrets scanner will search for secrets only in files tracked under git. Those untracked will not be scanned.

      --all-files            Scan ALL files. The default is to scan tracked
                               files only if under git.

By using --all-files, the scanner will search for secrets in all the files, regardless are tracked or untracked.

--staged-files

If you want to scan only the stagged files (i.e. only those modified but not committed yet) you can use the --staged-files and the scanner will only search for secrets in tracked but uncommitted files.

      --staged-files         Scan staged files (marked for commit), if under
                               git. Exits with error if not under git.

Preventing introduction of secrets (git hooks)

A git hook is a custom action that is launched when certain events occur in the Git version control system.

Git hooks can be used to prevent secrets to be committed to a git repository.

See Git Hooks with Xygeni for further help.

Secrets Verification

After scanning your codebase, Xygeni Secrets uses a proprietary verifier to determine if a secret is reachable, i.e. if it's valid.

All verifications, such as API calls, are done locally in your environment. No tokens or secrets are sent to Xygeni servers.

1. The verifier detects the service, such as Slack or AWS, that the secret is used for.

2. Xygeni Secrets makes an HTTP request using the secret.

3. If the HTTP request returns an HTTP status code of 200 or similar and some indication of valid access, the issue is flagged as verified.

The verification status is a key point into the severity of the secret. One verified secret logically cannot have the same severity as a not-verified.

A verified secret is a true statement (it's a certainty), Xygeni was able to use the secret to access the resource.

Instead, a not verified secret does not mean that it is not valid, it just states that Xygeni could not verify it by any reason. So be cautious about not-verified secrets.

There are some xygeni secrets parameters that are related to secrets verification :

      --no-verify            Ignore verifications that call to external
                               services.
      --only-verified        Report only verified secrets when there is a
                               verifier available.

Xygeni contains a directory ( conf/secrets ) where every secret detector has a yaml configuration file.

Into every detector config file, there is a section called verifier where its' specified the verifier class, some properties as well as the action.

Below is a sample with a generic verifier (ApiVerifier) that takes as parameters the API host and the authHeaderName to populate with the secret when validating it. Also, you can indicate the action to be done based on verification status.

verifier:
  className: com.depsdoctor.secrets.scanner.detector.verifier.ApiVerifier
  # action: do_nothing | increase_severity_when_verified | 
  #         decrease_severity_when_not_verified | ignore_when_not_verified
  action: decrease_severity_when_not_verified
  properties:
    host: gitlab.com/api/v4/projects
    authHeaderName: PRIVATE-TOKEN

Secrets Remediation

You should consider any sensitive data in commits with secrets as compromised, and the best option is to revoke or invalidate the secret, and remove it from the source code file. Remember that secrets may be removed from history in your projects, but not in other users' cloned or forked repositories.

To remediate the exposure of sensitive data, take the following steps depending on the nature of the secret identified:

1. False Positive or Expired Secret: If the secret is identified as a false positive or it has no impact, mark it as suppressed in the secret details.

2. Active Secret: If the secret is valid and active:

   • Revoke/Invalidate: Revoke or invalidate the secret, and remove it from the source code.

   • Remove from History: If it cannot be revoked, remove it from the repository history using tools like git filter-repo or BFG Repo-Cleaner. You may follow the procedure listed here for GitHub.

   • Check Logs: Inspect logs for any unauthorized, suspicious or unexpected activities from the time the secret was exposed.

Secrets Auto-Remediation

Xygeni provides mechanisms for automatic remediation of hardcoded secrets.

Please visit Secrets Auto-Remediation for further information.