Code Freeze: The Why and the How

In August and early September of 2022, JFrog helped a primary US telecommunication carrier host a series of sessions around developer best practices. Some 1,300 developers attended at least one session focused on package-specific examples covering Docker, Maven, Gradle, NPM, Nuget, and PyPi.

One of the intriguing takeaways from these best practices sessions was the uniform questions that arose in each session about conducting Code Freezes. 

Before we get started, here’s the code freeze meaning.

Code Freeze means the code is frozen, and there will not be any further modifications from the developers. After the code freeze, the developers should not change the code.

Purpose for Code Freezes

Every development team knows the meaning of a  code freeze.  While their reasons vary, there are two large categories most of them fall into – freezing for testing and freezing for business continuity. 

But, to be blunt, any code freeze is fundamentally at odds with the CI/CD concepts – Continuous Integration, Continuous Development/Deployment. However, there are some good reasons these freezes happen.

Testing freezes take place for a release as it is being finalized and going through the “extra” tests before release. These tests aren’t economical to run routinely, such as manual user-acceptance testing or anything with the word “manual” in the description. These tests don’t fit with CI/CD concepts that favor full automation. 

The most common business-focused code freeze is during peak sales times, such as Black Friday or Cyber Monday for retail. This code freeze preserves the customer experience–you don’t want anything but the most critical updates rolling out as customers try to purchase.

Protecting the Frozen Binary

So, it’s a given that code freezes are a part of CI/CD life. While the freeze is on, development still happens. This led to the question from multiple teams: What is the best way to ensure the “frozen” binary isn’t accidentally modified?

The general recommendation is to promote (move) a binary ready for the full battery of tests to a location with stricter RBAC controls than is typical. The variability and complexity come in to minimize the danger of an overwrite.

From a process point of view, ideally, you want this move to be restricted. At a minimum, log in as a different user. This user should only have permission to write to the target repository and read from the general repository. This limit discourages the common user of the profile.

Every developer eventually gets a hold of the “super user” for their group, usually while they’re doing something difficult and want to remove all possible permissions problems. This user’s existence is usually good, but the “freeze” repository is the exception. A super user shouldn’t have permission to write to this unique repository.

Also, automation can be the enemy of preservation here. A pipeline using the limited user RBAC to move or promote a binary to the “freeze” shouldn’t be able to be triggered accidentally by any other automation. There are two basic approaches here.

The first approach is not to hook this dedicated pipeline into any other automation. In this case, it should stand alone as a pipeline to be manually triggered by a limited number of seasoned individuals. The drawback here is this becomes manual, and the people can be the bottleneck (overlapping vacations or sick time may make them unavailable).

The second approach is to remove the user’s permission to write to the “freeze” repository. This approach becomes more sustainable, as any administrator can accomplish this function. If teams can self-serve at any level, this becomes viable. Jenkins pipelines allow teams to add and remove users and credentials. The user could be static, but the pipeline credentials can be removed.

Another approach is with sub-administration. Many companies have built infrastructure to allow for self-service tasks, like creating and removing users. Some systems provide direct sub-administration, where the RBAC target can be removed directly by developer groups (for example, the JFrog Projects approach).

With any of these approaches, the goal is to get a process in place to best conduct code freezes and, ideally move towards infrastructure-as-code. Because code freezes happen, a process around this will be prone to involve some handholding, at some level–and therefore, a risk area for mistakes.

Modern DevOps isn’t about perfection automation, it is actually about dealing with every element of a system eventually failing. The plan is to recover quickly. I encourage every group dealing with code freezes to take this mindset – think about where the processes you have in place can break down – and then start building your plan from there.