Securing Your Kubernetes Journey with ChartCenter

UPDATE: As of May 1, 2021 – ChartCenter central repository has been sunset and all features deprecated. For more information on the sunsetting of the centers read the Centers deprecation blog post

Adopting cloud native technologies like Kubernetes and Helm means your company’s operations can sail swiftly across the globe’s oceans to reach teams and customers. But there are dangers in the deep. 

With many components in Kubernetes, securing every dimension can be quite challenging and require a bit of learning curve. Let’s identify some important best practices that can help you to steer straight. 

We’ll also highlight some ways that JFrog’s free community service, ChartCenter, can reveal the risks that may lurk in the public apps you need to use, and help you to mitigate them.

Basic K8s Security Best Practices

To start, you should assure that all entry points for Kubernetes cluster resources are secure at their entry points. This includes the cloud project, network, cluster, api server, and namespaces, which need to be protected by setting up IAM, VPC, NAT, and  role-based access control (RBAC). Enabling this feature empowers you to set policies to permit or deny key operations based on the roles of individual users. You should secure this further by employing restricted namespaces to maintain isolation among different tenants or purpose. 

Once the Kubernetes clusters are secure, it’s vital to enable operational policies to ensure that the Kubernetes objects being represented by the YAMLs have key restrictions specified:

• Resource limits
• Security context
• Network policies

To further assure security, containers should only be run with an appropriate set of user-level permissions, and never be permitted to run as root.

Once your K8s system is running, the operations team should  have a process to continuously upgrade Kubernetes. In this way, any security issues will be patched and closed as quickly as they are recognized..

Helm Charts: What You See is What You Get

Once you have secured your K8s operating environment, you will need to be confident that the apps you deploy are safe to use. While Helm charts provide an efficient way of packaging Kubernetes apps, it’s important to understand their structure so you can be vigilant.

Most cloud native apps require multiple Docker images referenced by the Helm chart used to deploy them to K8s pods. These Docker images can be the main application as well as init containers and sidecar containers that are needed for the application to run. 

When we built JFrog ChartCenter, this was one of the problems we wanted to address by bringing visibility to polyglot dependencies of a Helm chart. For example, a Helm chart can point to many Docker images and sub-charts. These Docker images can further have multiple OS packages, UI packages, java, go and other application binaries. 

In the example below, for bitnami/postgres chart, a polyglot dependency graph is shown:

Each of these is an exploitable opportunity for a hacker. Helm charts and their supporting Docker images provided in good faith could be replaced with or redirected to intentionally vulnerable or malicious code.

To guard against this scenario, a Helm chart author can include a provenance file that can help assure the integrity of the chart being deployed. When provided, the provenance file includes a SHA256 signature of the chart package that can help detect any tampering. The entire body is signed using the OpenPGP algorithm, to further assure that the package was released from the known source.

JFrog ChartCenter offers a reliable source for Helm charts of publicly available K8s apps. When a Helm chart is included in ChartCenter, the availability of a provenance file is noted in the Chart Info tab.

When a provenance file is provided, it should be used with the helm install --verify to ensure that a breached chart won’t be deployed.

ChartCenter provides additional assurance of the chart version’s immutability. As a repository of Helm charts, not just a catalogue, the version held by ChartCenter is guaranteed to be always available and unchanged. This prevents even the chart’s author from introducing improper changes to a released version, ensuring that the chart version you use to deploy this week is the same version used last week, last month, or last year.

Guarding Against Critical CVEs

Even when you are confidently deploying the K8s app you expect, the app itself may have known vulnerabilities that a bad faith user might exploit. ChartCenter can help you to identify those risks.

In addition to bringing in visibility in terms of what’s deployed on a Kubernetes cluster via a Helm chart, we realized that it’s very important to display public CVEs associated with all the application bits that are being referred to by Helm charts. According to NIST,  the number and severity of CVEs registered on the National Vulnerabilities Database (NVD) is growing at rapid rate.

At the same time, it’s extremely hard to associate public CVE data on NVD with a component id that can be a Docker image, a Debian package, a Maven or other package. We realized that the only way to contain the growing impact of CVEs is to make security data at a Helm chart level easily accessible. 

ChartCenter runs continuous scans of 1.8M components in 12k+ Docker images being referred by 30k+ Helm chart versions. And this valuable information is what’s displayed on Chartcenter’s Security tab for every public Helm chart.

Along with a list of CVEs, an impact diagram can be observed by clicking on the graph icon.

Once vulnerability information is accessible, then it will help with prioritizing the uptake of security patches, or limiting the footprint (for example, by using a thin base layer, or dynamic tags while specifying base layers). 

Security Mitigation Notes

Once the critical information such as polyglot dependency graph, list of security issues, availability of provenance is accessible to secure the cloud native journey of users, another question to ask is what’s missing? 

Often, some of the detected CVEs can’t be exploited by the hackers. This may be due to the security or network policies set by the Helm chart, or a CVE might only be applicable under a different OS. In the cloud native ecosystem, there’s no current standard for producers to share security mitigation or advisory notes with consumers. In order to address this problem, we have proposed a standard that enables producers to provide security mitigation information with Helm charts. 

Once the author provides security mitigation information, it also influences when and what high CVEs are displayed on ChartCenter’s Security tab. The high CVEs are not shown by default without getting author’s consent. Once processed by ChartCenter, the referenced CVEs will indicate a mitigation note is available:

Click the icon to display the notes:

You can also download security mitigation notes for a given namespace and chart name. For example the following command will fetch the security mitigation file for the sample chart:

curl https://repo.chartcenter.io/rimusz/security-sample-chart/security-mitigation.yaml

Conclusion

Following some essential best practices can help you ensure that the cloud native Kubernetes apps you choose to run are secure.

With ChartCenter, you have additional tools to gain visibility into those apps, and have certainty that your business-critical operations are not open to unnecessary risks.