My source code is private, so why is hardcoding credentials in git considered a bad practice?

While private repositories offer some level of protection for your code they still do not have adequate protection to store information as sensitive as secrets.

Imagine if there was a plain text file with all your credit card numbers within it, you hopefully wouldn’t put this into the companies git repository. Secrets are just as sensitive.

A few things to consider when storing secrets in private repositories:

Source code is made to be duplicated and distributed, therefore lives in multiple places. Source code is a leaky asset and you never know where it is going to end up. It can be cloned to a compromised workstation or server, intentionally or accidentally published in whole or in part, uploaded to your website, released to a customer, pasted in Slack, or end up in your package manager or mobile application...
It would just take one compromised developer account to compromise all the secrets they have access to.
Hardcoded credentials make it impossible to know what secrets a developer accessed, and very difficult to rotate keys.

Why automate secrets scanning throughout the Software Development Life Cycle (SLDC)?

While DevOps, Continuous Integration and Continuous Delivery speed up software development, they can also significantly increase security risks.

“DevOps is rapid and requires lots of small, iterative changes. But this increases complexity and opens up a new set of security problems. With DevOps, existing security vulnerabilities can be magnified and manifest themselves in new ways. The speed of software creation can mean new vulnerabilities are created unseen by developers. The solution is to build security monitoring into the DevOps process from the start.”

Greg Day, RSA Conference Organizer (source)

Security now needs to be part of the SDLC from the start, and part of every incremental change. This concept is called Shifting Left, a development principle which states that security should move from the right (or end) of the SDLC to the left (the beginning). A great deal of automation is needed in order to be able to cope with running security checks at each incremental change.  Automation helps streamline the detection, alerting and remediation processes and workflows.

How does secrets detection compare with Static Application Security Testing (SAST)?

Secrets detection is often confused with SAST because both scan through static source code.

SAST is mostly testing control structure, input validation, error handling, etc. Such vulnerabilities like SQL injection vulnerabilities only express themselves the moment the code is deployed. Exposed secrets are unlike these vulnerabilities, because any secret reaching version control system must be considered compromised and requires immediate attention. This is true even if the code is never deployed.

Implementing secrets detection is not only about scanning the most actual version of your master branch before deployment. It is also about scanning through every single commit of your git history, covering every branch, even development or test ones.

To conclude, SAST is concerned only with the current version of a project, the version that is going to be deployed, whereas secrets detection is concerned with the entire history of the project.

What are git hooks?

Git hooks are scripts that are triggered by certain actions in the software development process, like committing or pushing. By automatically pointing out issues in code, they allow reviewers not to waste time on mistakes that can be easily diagnosed by a machine.

There are client-side hooks, that execute locally on the developers’ workstation, and server-side hooks, that execute on the centralized version control system.

Examples of client-side hooks:

Pre-commit
Post-commit
Pre-push

Examples of server-side hooks:

Pre-receive
Post-receive

What is a pre-commit hook?

"The pre-commit hook is run first when committing, before you even type in a commit message. It’s used to inspect the snapshot that’s about to be committed, to see if you’ve forgotten something, to make sure tests run, or to examine whatever you need to inspect in the code.

Exiting non-zero from this hook aborts the commit, although you can bypass it with git commit --no-verify. You can do things like check for code style (run lint or something equivalent), check for trailing whitespace, or check for appropriate documentation on new methods."

Source: git-scm

What is a pre-receive hook?

A pre-receive hook is a script that runs on the server. It performs checks on the content of the push; if it exits non-zero, the push is rejected. You can use this hook to do things like prevent a PR author from merging their own changes, or require commit messages to follow some specific guidelines.

Be careful when using pre-receive hooks as they are blocking: if the checks don’t pass, the server is not updated.

What is a post-receive hook?

"The post-receive hook runs after the entire process of pushing code to the server is completed and can be used to update other services or notify users. Examples include emailing a list, notifying a continuous integration server, or updating a ticket-tracking system – you can even parse the commit messages to see if any tickets need to be opened, modified, or closed. This script can’t stop the push process, but the client doesn’t disconnect until it has completed, so be careful if you try to do anything that may take a long time."

Source: git-scm

Unlike the pre-receive hook, post-receive is non-blocking.

Where in the DevOps pipeline to implement automated secrets scanning? Client-side or server-side?

The earlier a security vulnerability is uncovered, the less costly it is to correct. Hardcoded secrets are no exceptions. If the secret is uncovered after the secret reaches centralized version control server-side, it must be considered compromised, which requires rotating (revoking and redistributing) the exposed credential. This operation can be complex and typically involves multiple stakeholders.

Client-side secrets detection early in the software development process is a nice-to-have as it will prevent secrets entering the VCS earlier.

Server-side secrets detection is a must have:

 Server-side is where the ultimate threat lies. From the server, the code can uncontrollably spread in a lot of different places, with the hardcoded secrets in it.
Implementing client-side hooks on an organization level is hard. This is something we have heard many application security professionals claim they are not confident to do, due to the difficulty to deploy and update this on every developer’s workstation.
Client-side hooks can and must be easy to bypass (remember secret detection is probabilistic). Usage of client side hooks largely comes down to the individual developers’ responsibility. Hence the need to have visibility over the later stages.

Should secrets detection be blocking or non-blocking in the SDLC?

From our experience, when trying to impose rules that are too constraining, people will bend them, often in an effort to collaborate better and do their job. Security must not be a blocker. It should allow flexibility and enable information to flow, yet enable visibility and control.

On one hand, security measures will be bypassed, sometimes for the worst. But on the other hand, it is also good sometimes that the developer can take the responsibility to bypass them.

Because even the best algorithms can fail and need human judgement. Secrets detection is probabilistic: algorithms achieve a tradeoff between not raising false alerts and not missing keys.