Best Practices for CI/CD Security

For organizations that have embraced DevOps, the CI/CD pipeline is the foundation of the software delivery workflow. It facilitates the continuous movement of code from development to testing to deployment, and it allows the various stakeholders in the software delivery process — developers, test engineers, IT operations engineers and more — to collaborate.

Given the centrality of the CI/CD pipeline to DevOps, CI/CD security should be a priority for DevOps teams. Insufficient CI/CD security can result in risks such as the insertion of malware into application code as it passes down the CI/CD pipeline or the exposure of sensitive data that is managed within the pipeline.

To help prevent problems like these, this article discusses strategies for securing the CI/CD pipeline at all stages.

Establish access controls for CI/CD tools

It’s common for DevOps teams to think of CI/CD as a process that all stakeholders “own” collectively. After all, sharing visibility into and responsibility for the software delivery process between developers and IT operations teams is part of the core focus of DevOps.

As a result, it may seem logical to allow all team members to access all tools and resources within the CI/CD pipeline. But this is a mistake. It’s possible to share visibility and responsibility without sharing access.

Instead of granting all team members full access to all CI/CD tools, data and environments, access should be assigned on the basis of roles. Only developers should be able to manage CI servers and source code management tools, for example. For their part, IT operations teams typically only need access to management tools that are used for production environments. They don’t need to control dev/test environments or pre-deployment tools.

Ensure environment parity across the pipeline

Environment parity means that all environments within the CI/CD pipeline, such as dev/test and production, are configured in the same way.

From a security perspective, environment parity is important because it ensures that the conditions under which software is tested match the conditions under which it will run in production. Without environment parity, teams run the risk of failing to detect a security issue during testing because the issue only exists under a certain environment configuration. It may happen only with a certain OS version, for example, or with a particular version of a software library.

A simple means of achieving environment parity is to host software inside containers at all stages of the CI/CD pipeline. Although containers don’t guarantee complete environment parity (different versions of the container runtime could create vulnerabilities in one environment that don’t exist in another, for example), they do abstract applications from the host environment in such a way that most environment variables on the host don’t impact the way the application performs. Thus, if you test your application inside a container and then deploy the application to production using the same container, you can have higher confidence that your tests have adequately covered potential security risks.

Don’t store secrets in source code

DevOps teams often use a variety of passwords, access keys and other secrets to manage the CI/CD pipeline. They need account credentials to access CI servers, for example, and SSH keys to log into servers that host their applications.

It can be tempting to hard-code this data directly into configuration files or source code, where it is easy to access repeatedly. And if the secrets are used only for testing environments or other pre-production stages of the pipeline, engineers may assume that there is little risk in storing the data in this way, because it can’t be used to exploit public-facing applications that have been deployed into production.

The reality is that secrets stored inside code are simply not secure. They can be leaked to malicious parties, who may use them to gain unauthorized access to CI/CD resources. Even if the secrets apply only to the development or testing phases of the CI/CD pipeline, attackers could use them to do things like inject malicious code into the application. If left undetected, the malicious code will be pushed into production.

Instead of storing secrets in configuration code or source code, use a secrets manager, which provides a secure means of storing secrets of all types and configuring access to them based on team members’ roles.

Test and monitor all CI/CD resources

Most teams know that they need to run security tests on their applications. But what they may overlook is the importance of also testing other resources within the CI/CD pipeline — such as build and deployment scripts, or Infrastructure-as-Code (IaC) files — for security vulnerabilities. Insecure configurations are another source of unauthorized access that may lead to problems like the insertion of malicious code into an application.

Likewise, rather than only monitoring applications within production environments for signs of security problems, teams should deploy automated testing tools in the dev/test environment as well — which is one example of a practice known as shift-left security.

Prepare for rollbacks

Sometimes, you may discover a security issue that affects an application you have already deployed. Performing a rollback, which means reverting to an earlier version of the application that you know to be secure, is often the fastest way to eliminate the risk until developers can fix the underlying security problem.

For this reason, it’s a CI/CD security best practice to ensure that you can roll back every release quickly. Keeping artifacts from older versions of your application on hand helps to achieve this goal. So do deployment scripts or release automation tools that can be configured to re-deploy a previous version of the application, as opposed to only deploying new releases.

Secure the software supply chain

The greatest security risks to your CI/CD pipeline may emerge not from code or configurations that you write yourself, but from third-party dependencies, such as open source libraries that you import into an application.

If left unaddressed, security vulnerabilities that affect third-party resources can lead to what are known as software supply chain attacks. Software Composition Analysis (SCA) tools, which scan your application source code for third-party dependencies and identify those that are insecure, can help prevent this risk. So can package scanners, which check packaged applications or container images for components that are subject to known vulnerabilities.

Finally, consider using signed code and artifacts within your CI/CD pipeline. Signing confirms the origins of a component, giving you confidence that the resources you use within CI/CD processes came from trusted sources.