What are secrets in the software development world?

In the common language, a secret can be any sensitive data that we want to keep private. When discussing secrets in the context of software development, secrets generally refer to digital authentication credentials that grant access to systems or data. These are most commonly API keys, usernames and passwords, or security certificates.

Secrets exist in the context of applications that are no longer standalone monoliths. Applications nowadays rely on thousands of independent building blocks: cloud infrastructure, databases, SaaS components such as Stripe, Slack, HubSpot… 

Secrets are what tie together these different building blocks of a single application by creating a secure connection between each component.

What is secret sprawl?

Secret sprawl is the unwanted distribution of secrets like API keys and credentials through multiple systems. 

In modern software development, secrets are regularly used by developers and applications. As a result they often get shared through different services like Slack or email and can be stored in multiple locations including different machines, git repositories or inside company wikis. This is a phenomenon we call secret sprawl.

What are some of the best practices to securely manage secrets like API keys?

Storing and managing secrets like API keys and other credentials can be challenging. Even the most careful policies can sometimes be circumvented in exchange for convenience. We have put together a helpful cheat sheet and article explaining the best practices for managing secrets.

As a minimum here are some points you should consider:

Never store unencrypted secrets in git repositories
Avoid git add * commands on git
Add sensitive files in .gitignore
Don’t rely on code reviews to discover secrets
Use automated secrets scanning on repositories
Don’t share your secrets unencrypted in messaging systems like Slack
Store secrets safely (method to be used depends on every use case)
Default to minimal permission scope for APIs
Whitelist IP addresses where appropriate
Use short-lived secrets

What are the threats associated with secret sprawl?

When secrets are sprawled through multiple systems it increases what is referred to as the ‘attack surface’. This is the amount of points where an unauthorized user could gain access to your systems or data. In the case of secret sprawl, each time a secret enters another system it is another point where an attacker could gain access to your secrets.

Most internal systems are not an appropriate place to store sensitive information, even if those systems are private. No company wants credit card numbers in plaintext in databases, PII in application logs, bank account credentials in a Google Doc. Secrets benefit from the same kind of protective measures.

As a general security principle, where feasible, data should remain safe even if it leaves the devices, systems, infrastructure or networks that are under organizations’ control, or if they are compromised. This helps prevent credential stealing, which is a well-known adversary technique described in the MITRE ATT&CK framework:

“ Adversaries may search local file systems and remote file shares for files containing passwords. These can be files created by users to store their own credentials, shared credential stores for a group of individuals, configuration files containing passwords for a system or service, or source code/ binary files containing embedded passwords. ”

Secrets accessed by malicious threat actors can lead to information leakage and allow lateral movement or privilege escalation, as secrets very often lead to other secrets. Furthermore, once an attacker has the credentials to operate like a valid user, it is extremely difficult to detect the abuse and the threat can become persistent.

What are some of the biggest challenges associated with secret sprawl?

Because secrets tie together each component of an application, developers and ops need access to these secrets to build, connect, test and deploy applications. The result is that secrets need to be both tightly wrapped and also widely distributed.

Enforcing good security practices at the organization level is hard. Developers today can have large turnovers inside companies, be spread through many different teams and geographies. They have a growing number of technologies they must master and are under hard pressure due to shortened release cycles. This makes secret management very complicated and an ever changing challenge.

This is further complicated when VCS like git are introduced because secrets can be buried deep inside the history. The actual version of source code might look clean, while the history might present credentials that were added than later removed. Any secret reaching the Version Control System must be considered compromised and the presence of valid secrets in the git history presents a threat!