Mitigating Image Integrity Violations: A Real-World Example in Runtime Environments

In the never-ending quest to speed up software release cycles, ensuring the security and integrity of application artifacts has never been more critical. As applications are continuously built, tested, and deployed, every element of the software pipeline—from source code to container images—needs to be trusted and verifiable.

A key aspect of maintaining this trust is image integrity protection and validation. This crucial process ensures that only authorized, untampered images are deployed, safeguarding production environments from malicious alterations.

JFrog Runtime introduces robust image integrity protection mechanisms, empowering development teams to confidently manage their software artifacts while securing the end-to-end release process. In this blog post, we’ll explore why image integrity validation is vital to the SDLC and how JFrog Runtime enhances this protection.

Why Image Integrity Validation is Critical

In modern software development, the continuous integration and deployment of containerized applications introduce new layers of complexity and security challenges. Container images, which package application code and its dependencies, traverse multiple stages of the pipeline. They are built by different teams, stored in various registries, and finally deployed across production environments. Without proper validation, these images are susceptible to tampering, accidental corruption, or malicious alterations, increasing the risk of deploying compromised software that could lead to severe security incidents like unauthorized access, data breaches, or infrastructure failures.

Image integrity validation ensures that the image used in production is exactly the same as the one approved during development. By preventing the deployment of altered or outdated images, it safeguards against unauthorized changes and the introduction of vulnerabilities. Incorporating clear data lineage into the SDLC supports these efforts. Tracking the provenance of container images—from development to deployment—gives teams complete visibility into every change, version, and update, allowing organizations to pinpoint the exact source of any integrity issues and providing a clear audit trail.

JFrog Runtime continuously validates data lineage and image integrity, ensuring no unauthorized changes slip into production environments. Its real-time monitoring of running containers detects discrepancies between the deployed instances and the images stored in the repository. This capability empowers teams to identify and address issues immediately, maintaining the security and resilience of the software environment.

Security Landscape Affecting Container Image Integrity

In containerized environments, particularly in Kubernetes deployments, several security challenges can compromise the integrity of container images. The threat landscape is complex, with both external adversaries and inadvertent internal actions posing risks. Attackers can exploit vulnerabilities in the supply chain, compromised registries, or local caches to introduce malicious code or unauthorized changes into running environments.

Some of the main potential security threats include:

• Man-in-the-middle (MITM) attacks – A particularly dangerous attack vector is where an attacker intercepts image transmission between the registry and the environment, injecting a malicious image that replaces the legitimate one at runtime.
• Registry spoofing – In this scenario, attackers attempt to access a private registry or spoof a public one, tricking Kubernetes into pulling compromised images.
• Cache poisoning – Injection of malicious images into the local cache can cause Kubernetes to deploy defective images instead of the intended ones from the correct registry. This can happen without detection, as the tags and metadata may still match the legitimate image, creating a false sense of security.
• Internal errors or misconfigurations – Often lead to the deployment of images from untrusted registries or outdated sources, increasing the risk of running images that differ from those intended for deployment. These issues occur frequently and mostly stem from incorrect registry configurations, manual interventions, or overlooked updates. In fact, according to a recent survey, 28% of security professionals consider misconfigurations as the number one security risk in container environments.
• Moving Tags – Tags such as :latest, :stable, or :prod are particularly vulnerable to integrity issues. These tags remain unchanged even as the underlying image is updated with newer versions. This can lead to various issues. In Kubernetes environments, this can cause automatic, unapproved updates when the system pulls the latest image without explicit approval. Conversely, if the system is configured to avoid automatic updates, it can lead to integrity violations due to a lack of clear understanding of which image is actually in use.

To mitigate these risks, continuous validation and monitoring are essential to ensure that only trusted, verified images are deployed and running in production environments.

Securing Container Image Integrity

The integrity of a container image relies on two critical factors: ensuring the image’s source is trusted and verifying the binary integrity of the image itself.

JFrog Runtime provides real-time insights into images deployed from untrusted registries. If an image is deployed from an untrusted source, it is flagged with a corresponding indicator within the JFrog Platform. This enables the platform operations team or the Site Reliability Engineers to take corrective action, such as ensuring that images are deployed from a trusted registry.

By deploying all production images from the JFrog Artifactory image registry, the platform operations team can ensure that each image has passed through the organization’s defined security checks and policies. After confirming the source registry is trusted, JFrog Runtime checks for binary differences between the running container images and the original images. Any image exhibiting integrity issues is flagged, enabling the platform operations team to quickly detect and resolve these issues, ensuring the correct images are running.

JFrog Runtime’s real-time monitoring not only enhances security but also significantly reduces the Mean Time to Remediate (MTTR) for integrity violations. By quickly identifying discrepancies between running containers and approved images, the platform enables immediate corrective action, allowing teams to quickly address unauthorized changes. These real-time indicators, provided during runtime, significantly reduce remediation time, enabling faster responses to security or configuration issues.

The JFrog Platform, with built-in integration between JFrog Runtime and JFrog Artifactory, streamlines risk identification and remediation efforts by linking the runtime environment directly to the image registry. This integration allows for real-time detection of image integrity issues, eliminating the need to switch between systems. Teams can easily trace problematic images back to their source and owner in Artifactory, accelerating the identification and remediation process.

This seamless integration increases security and boosts operational efficiency, offering end-to-end visibility and control over image management and risk remediation.

Mitigating Image Integrity Violations: A Real-World Example

To illustrate how JFrog Runtime actively safeguards image integrity, let’s consider a real-world example.

Examining the Live Assessment screen in Runtime, we can see the list of images running in Runtime. The list is sorted by risk severity, presenting the riskiest image at the top.

JFrog Runtime Live Assessment Screen presenting the running image with the integrity violation risk

In this scenario, the green-burrito image has been flagged with an integrity violation risk, indicating that the image running in the environment is different from the one stored in the JFrog Artifactory registry.

Although integrity violations can result from malicious attacks, they are more often caused by internal factors, such as developer oversights, misconfigurations, or mistakes during the development process.

The integrity violation in this case occurred due to the use of the :latest tag. This tag is unreliable as it doesn’t guarantee consistency, often leading to discrepancies between the running image and the expected version in the registry. In this instance, a newer version was pushed to Artifactory under the :latest tag, but the running workload failed to fetch the update, continuing to use the outdated image.

JFrog Runtime Live Assessment indicates the latest tag is running in the production environment

After inspecting the workload configuration, we found that the imagePullPolicy was set to IfNotPresent. This setting means the workload only pulls the image from the registry if it’s not already present locally. As a result, the workload didn’t fetch the updated image with the :latest tag and continued running the outdated version, leading to the integrity violation.

imagePullPolicy configuration in the workloads resource configuration

To resolve this integrity issue, we updated the imagePullPolicy to Always in the workload configuration and restarted the deployment.

Setting imagePullPolicy: Always ensures that the workload consistently pulls the most recent version of the image from the registry whenever it starts, thereby preventing outdated images from running and ensuring alignment with the expected version. However, it’s important to note that this setting may not always be the best approach and should be carefully evaluated in real-world scenarios, as it permits automatic, unapproved updates.

imagePullPolicy modification in the running workload and inspection using k9s (a Kubernetes management tool)

The updated Live Assessment screen below confirms that the integrity violation has been resolved. The green-burrito image is now running the expected version, demonstrating the effectiveness of the remediation. This highlights the importance of using fixed tags and appropriate imagePullPolicy settings to maintain image consistency in runtime environments.

imagePullPolicy modification in the running workload and inspection using k9s

This example demonstrates the essential role of image integrity validation in maintaining a secure runtime environment. By implementing robust validation practices and enforcing consistent configurations, teams can minimize the risk of deploying outdated or unauthorized images, ensuring that only trusted and verified artifacts are running in their production environments.

Take the Next Step: Secure Your Software Supply Chain with JFrog Runtime

If your organization wants to enhance security by guaranteeing image integrity while increasing development efficiency, we invite you to schedule a live demo of JFrog Runtime to discover how the  JFrog Platform can secure your software supply chain.