The CI/CD Secrets Ecosystem & the basics of Kubernetes Secrets Management

What are Kubernetes secrets? 

Security of enterprise data is extremely important when it comes to storing and protecting Certificates, API tokens, SSH keys, passwords, etc. Kubernetes Secrets are secure objects and sensitive data stored in vaults in a centralized and secured way. Usually, such information is saved in code repos like Git, BitBucket, DBs, or even local files. However, while using secrets, make sure you don’t need to save your confidential information in such a manner inside your application code. Secrets are stored in etcd, a datastore of Kubernetes. There are various ways to store them but the most common one is by loading them into environment variables.

Another way is by using volume mounts. Like;

What is secret management?

Secret management simply put is nothing but password management. It steadily manages the creation, alternation, withdrawal, and storage of digital authorization credentials.

Why is Secret management crucial?

As secrets consist of highly classified information, they must be kept safe in transition and at rest to mitigate the risks. Secrets usually include:

• Auto-generated passwords
• SSH Keys
• Database and other system-to-system passwords.
• Private certificates for secure communication
• API and other application tokens/credentials 
• User password

But with the diverse and fragmented IT ecosystem and its complexities, it’s a challenge to store, transmit and audit Kubernetes secrets safely. Often, enterprises are unaware of when their secrets get breached. It is observed that these data breaches are mainly caused by human errors. In addition, we also have to consider the outages and expired certificates that can stir the users, resulting in organizations losing their reputation.

Common secret management challenges: 

Cross-cloud secrets sprawl: Incomplete visibility and awareness

As technology, resources, accounts, and applications change constantly, the locations of these secrets shift as well. Further, several applications running across multiple clouds demand a centralized management system. This aspect makes it difficult for organizations to have clear visibility and challenging to manage and audit the secrets effectively.

• Scalability and Orchestration

It is important to auto-scale secret management infrastructure and ensures that secret management tools orchestrate well with other tool stacks (TeamCity, GitLab, Jenkins, CircleCI, etc) when services operate at a large scale across clouds. However, this becomes quite challenging when scaled across multiple clouds, as each cloud has its own set of workflows in place. 

• No revocation and rotation

Exposure to secrets can lead to data breaching. Therefore, it is a crucial requirement to revoke user credentials from time to time to avoid this. For example, revoking secrets should be a normal response to: 

– An employee’s resignation

– The termination of an agreement with a third-party vendor, or

– Failed authorization attempts

But, many times, organizations do not pay attention to the above scenarios.

Moreover, many security standards require changing the passwords at a fixed interval, and by not paying heed to them, organizations risk their data. Rotation is a vital step in the lifetime of a secret, as regular rotation makes sure no stolen keys or passwords can be used for an extended period. Thus, short secret lifetimes result in a reduction of the surface area of attack. 

• Reusing secrets

Employees tend to reuse the secrets for various accounts or applications to save time. While that helps them remember the secrets easily, reusability brings in a higher risk. If one secret gets breached, all the linked accounts and applications have a chance of getting misused. 

Pointers to look out for when managing secrets

Secrets management handles secrets effectively in a centralized manner throughout the secrets’ lifecycle. While we have seen the challenges of secret management, let us check out some of the ways used to bring solutions to them by keeping your organization’s secrets truly a secret.

1. Create a secret management policy

As seen at the beginning of this article, there can be many glitches in organizations’ managing secrets. The Forrester DevOps report recently revealed that 57% of organizations suffered security incidents related to exposed secrets in DevOps. The number of secrets scales up with the growth of an IT infrastructure, making it difficult to protect them. Thus, building a secret management policy with strict Dos and Don’ts is necessary to avoid them all. Listed below are some of the points to consider while creating a policy: 

Define strict rules for the structure of Kubernetes secrets management

1. Removal of hard-coded credentials 
2. Regular audits
3. Access control lists
4. Rotation and revocation
5. Centralize your secrets
6. Keep encrypted keys
7. Dynamic secrets

2. Automate Secret Management Processes across Clouds

Manual tasks often create a possibility of an error. This is the reason why it’s important to put an end to manual tasks from secret management and automate deployments on a reliable secret management software. The best way to avoid errors is by using tools for key generation, rotation, revocation, storage, and transmission of secrets. Unique practices are required to manage vault secrets and privileged access while working in CI/CD. Kubernetes secrets management in a CI/CD environment is expensive and error-prone if not managed perfectly. But using the available automation tools, one can handle cloud Kubernetes secrets management centrally in a continuous environment that can be consumed by other CI/CD systems.

3. Enforce privilege management 

Various users and applications with access to Kubernetes secrets put the organizations’ critical data, services, and resources at risk. One way to avoid this is to ensure the secret management policy is followed by setting up session monitoring and reviewing audit logs. Moreover, make sure no application or personnel has more privileges than needed.

Let us have a look at some of the most popular and ubiquitous secret management solutions tools:

• HashiCorp Vault

An open-source secret management tool that helps protect sensitive data and manage secrets. HashiCorp stores all kinds of secrets and manages controls people have on the data. The organizations can rotate the sensitive data and revoke the access at any time, keeping the credentials safe and secured. It provides static secrets, dynamic secrets, and automation for enterprises, and even integration with various other applications. 

• AWS Secret Manager

The AWS secret manager tool helps protect secrets while accessing applications, services, or resources. It also enables you to rotate, manage and retrieve the API keys, credentials, and passwords throughout a secret’s lifecycle. 

• Microsoft Azure Vault

Another popular name in the field of Secret management is Microsoft Azure Key Vault. By using this Key Vault, you don’t need to provision, configure, patch, or maintain hardware security models and critical management software. Instead, simply maintain your keys and provide access to anyone or any app, to keep your data safe and in your control.  

• Kubernetes Secrets Manager

Kubernetes Secret Manager stores and carries sensitive information for running on any web service. Moreover, it is fairly simple to integrate Kubernetes secret with already running services. Hence, Kubernetes can easily be integrated with third-party cloud solutions and open-source tools. 

What are dynamic secret injections?

The dynamic secret is a credential that’s not directly consumed from the vault. The credentials are created initially by a user and leased out by the vault service. Meaning, that database passwords are not created per se but put into the Vault which acts as a gatekeeper for the Kubernetes secrets, providing passwords on demand. Think of it as a ticket to a movie show that expires once the movie has ended. In the same way, Vault here acts as the watchman in verifying the ticket and authenticates the external system to access the database/app etc.

Why is this useful? We need this as otherwise, applications do an utterly appalling job keeping anything a secret. Most organizations have credentials hardcoded in source code scattered throughout the configuration files and tools and stored in plaintext. Vaults provide them a central place to keep credentials encrypted, audited, and visible only to the authorized personnel. But there are two main challenges here:

a) Centralization – when an organization stores sensitive information such as passwords, API keys, SSH tokens, etc., in many different places, it becomes problematic to keep them secured. Thus, it is essential to manage secrets efficiently from the beginning so that you do not end up with secrets sprawled in various places. Achieving centralization is critical here, as secrets sprawl increases with the growth of an organization’s IT infrastructure. 

b) Sharing common credentials in services – Most organizations make the mistake of defining the same credentials for many services and are unaware of the pitfalls they might face if these secrets are leaked. It becomes difficult to identify the source of the leaked credentials and update them in such cases. A workaround can be to disable just the compromised credentials rather than changing the root credentials. Since secret sharing doesn’t expose the actual secret, the scope of the attack is reduced, and organizations can disable compromised credentials. 

Dynamic secrets come as a savior to overcome all these challenges.

Dynamic secrets, as the name suggests, are dynamic and not static. Thus, it is unique to every client and generated on demand. Vault associates this dynamic secret with a lease and automatically writes out the credentials once the lease expires. So, stack up all the certificates with an in-built vault to prevent secrets spread from your applications. Then, inject the Kubernetes secrets dynamically into CI/CD pipelines; rest assured that your secrets are safe!

In the context of ozone’s dynamic secret injection for CI/CD, the application’s runtime requests the secret and ozone uses a vault to supply secrets to the CI/CD runtime. Once the runtime ceases to exist, Ozone cleans up the secrets. 

Simplifying CI/CD Governance and Security with OZONE

OZONE, the modern CI/CD platform, has a highly appreciated module for continuous security for DevOps, which assists you to govern with control and secure the organizations’ sensitive details without hassle. Our team at Ozone Cloud Inc has carefully designed the Ozone CI/CD platform that addresses the need for DevSecOps in your IT infrastructure. DevSecOps, a combination of Development, Security, and Operations programs, amalgamates security at every phase of the Software development lifecycle. So, let us check out how OZONE enhances CI/CD security by dynamically injecting secrets across DevOps stages during the runtime.

• Kubernetes secrets management is an essential part of DevSecOps. Ozone injects secrets right into the pipeline across DevOps stages, thus reducing manual configurations.
• Shift-left DevSecOps with runtime scans to trace vulnerabilities long before you hit production
• Enterprises can leverage Single Sign-on (SSO) authentication with multiple protocols like  OAuth, LDAP, AD, and more for unifying people and processes on the Ozone platform
• Ozone offers allowed and secure communications to private clusters for deployments, monitoring, and logging across public and private clouds
• Enhanced security with automation helps make DevOps’ secure by design.’
• ML-based deployment verifications help eliminate human errors and overheads while monitoring your cross-cloud microservice deployments and automating roll-backs should there be anomalies
• Audit trails and traceability with intuitive dashboards for delivering meaningful insights

Conclusion

Secret management and injecting dynamic secrets at various levels of DevOps is an essential aspect of the Software Development lifecycle. Most organizations miss out on paying attention to these aspects. In static secrets, all the entities who need to access the service would share the same set of credentials, and thus, in case of a breach, there would be a need to change the secret for each entity, which will be pretty cumbersome. Whereas in the case of dynamic secrets, we can quickly identify the stumbling block and change the secret only for the breached entity. If you are looking for one such solution which can help with your Kubernetes secrets management, reach us to know more about our preeminent tool, OZONE.