Power Up Helm Charts: Using Kustomize to Manage Kubernetes Deployments

Operations teams need to automate deployment of applications, and Helm is the industry-standard solution for deploying to Kubernetes. Like many software vendors, we provide Helm charts for installing Artifactory and other JFrog DevOps platform products, designed for the standard/recommended configurations most teams will need. These templates provide customers a limited set of configuration options.

If you need to customize your Artifactory, Xray, or other deployments in ways of your own, you might fork the chart to create your own custom version. But once you do, each time JFrog updates its Helm chart, your custom version becomes out of sync and obsolete. To keep your version up-to-date, you would need to reintegrate for every update.

How can you customize a Helm chart without forking? That’s exactly what Google created Kustomize for. In this post — and also in an upcoming webinar — we’ll show you how you can use Kustomize overlays to perform custom deployments while always using the latest Helm chart version from your vendor.

Templates vs Overlays

A template is a form that has placeholders that an automated process will parse to replace them with values. Designed to perform a specific function, it marks the places where you must provide the specifics. If you’ve ever played “Mad Libs” this “fill-in-the-blanks” process will be familiar.

As a developer, you’ll recognize that templates and values are like macros and their variables, or subroutines and their parameters. 

An overlay is a set of replacement strings. Blocks of text in the original file are entirely replaced with new blocks of text.

What’s the difference? 

  • A template needs to be carefully prepared to demand specific information in key places. When you use a template, you’re restricted to changing only those elements the template makes available. 
  • An overlay doesn’t require the original file to be prepared in any way. You can replace any part in its entirety.

HelloWorld Helm Chart

Since Artifactory’s Helm charts are quite advanced, let’s use a very simple example. Here, the template permits an argument for the company name.

$ cat templates/pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  restartPolicy: Never
  containers:
  - name: hello
    image: alpine
    env:
    command: ["/bin/sh","-c"]
    args: ["/bin/echo Hello! My company name is {{ .Values.companyName}}"]

 

The values for the template’s arguments are in the file values.yaml.

$ cat values.yaml
companyName: ABC Company

 

Let’s install and see how it works.

$ Helm install helloworld .
NAME: helloworld
LAST DEPLOYED: Mon May 18 16:53:14 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

$ kubectl logs -f helloworld
Hello! My company name is ABC Company

 

Neat! This worked well, and we shipped it to customers. But one customer has a unique requirement, to add employee name and department name to the output to look like this: 

My name is John. I work for the Accounting department. Our company name is ABC Company.

Okay, pretty easy, they could fork the Helm chart and change the Helm template as follows:

    args: ["/bin/echo My name is {{ .Values.employeeName}}. I work for {{ .Values.employeeDepartment}} department. Our company name is {{.Values.companyName}}"]

 

Then they would add new values in the values.yaml file.

$ cat values.yaml
employeeName: Gary
employeeDepartment: Marketing
companyName: ABC Company

 

But, as we noted, their fork is now out of sync with the original. This is where Kustomize comes to the rescue.

Overlaying with Kustomize

Kustomize enables you to  overlay your own “kustomizations” in yaml files. In our example, the customer can “kustomize” to their needs without creating a private, unmaintainable fork of charts.

First the customer will create a kustomization.yaml file.

patchesJson6902:
- target:
    version: v1
    kind: Pod
    name: helloworld
  patch: |-
    - op: replace
      path: /spec/containers/0/args
      value: ["/bin/echo My name is {{ .Values.employeeName}}. I work for {{ .Values.employeeDepartment}} department. Our company name is {{ .Values.companyName}}"]
resources:
- templates/pod.yaml

 

Now we can direct kustomize to apply our overlay.

$ mkdir templates_new
$ kustomize build -o templates_new

$ cat templates_new/~g_v1_pod_helloworld.yaml
apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
  - args:
    - /bin/echo My name is {{ .Values.employeeName}}. I work for {{ .Values.employeeDepartment}}
      department. Our company name is {{ .Values.companyName}}
    command:
    - /bin/sh
    - -c
    image: alpine
    name: hello
  restartPolicy: Never

 

We will first replace the original template with our new template, then install it with Helm and verify.

$ mv templates templates_old

$ mv templates_new/ templates

$ Helm delete helloworld
release "helloworld" uninstalled

$ Helm install helloworld .
NAME: helloworld
LAST DEPLOYED: Tue May 19 14:27:18 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

$ kubectl logs -f helloworld
My name is Gary. I work for Marketing department. Our company name is ABC Company

 

Great! This matches our customer’s requirement.

Template First, Overlay Next

In the prior example, we used Kustomize to modify our Helm template to accept new values, then used that version with helm install to deploy the app. 

But there are scenarios where that sequence isn’t ideal. Instead, you may wish to render the chart template first locally, then apply your Kustomize overlay when you deploy the app.

This may work best when you need to deploy the same app to multiple environments, but with cross-cutting concerns such as labels, security, or metering. For example you may have different combinations of requirements for each environment:

Environment labels security metering
dev Yes No No
test Yes No Yes
production Yes Yes Yes

 

Similarly, you may need to adjust ports or access for each of these environments. In these scenarios, it may be more flexible to apply a different Kustomize overlay to the same rendered Helm chart for each environment.

For this purpose, the kubectl command line utility provides the apply -k option. This facility applies Kustomize on charts using kustomization.yaml files in the directories.

To start, let’s render the Helm chart locally using the helm template command. This outputs a YAML file with all values resolved, which we capture to a local file.

$ mkdir templates_new
$ helm template . > templates_new/pod.yaml

$ $ cat templates_new/pod.yaml
---
# Source: helloworld/templates/~g_v1_pod_helloworld.yaml
apiVersion: v1
kind: Pod
metadata:
  name: helloworld
spec:
  containers:
  - args:
    - /bin/echo My name is Gary. I work for Marketing
      department. Our company name is ABC Company
    command:
    - /bin/sh
    - -c
    env: null
    image: alpine
    name: hello
  restartPolicy: Never

 

Create a new kustomization file to add labels to our pod as follows:

$ cat templates_new/kustomization.yaml
commonLabels:
  app: helloworld
resources:
- templates_new/pod.yaml

 

Now  let’s use kubectl apply -k to install our chart with new labels:

$ helm delete helloworld

$ kubectl apply -k templates_new/.
pod/helloworld created

$ kubectl get pods
NAME         READY   STATUS      RESTARTS   AGE
helloworld   0/1     Completed   0          10s

$ kubectl describe pod helloworld
Name:         helloworld
Namespace:    default
Priority:     0
Node:         docker-desktop/192.168.65.3
Start Time:   Mon, 22 Jun 2020 16:22:11 -0700
Labels:       app=helloworld
Annotations:  Status:  Succeeded
...
...

 

Success! Now we are able to take any vendor’s Helm chart and add our own‘kustomizations on top while continuing to take updates from upstream charts.

Dive Deeper

JFrog’s Helm charts are quite advanced, and we’ve often had to maintain multiple versions for our internal use. Using Kustomize has helped us to avoid manual editing of charts, and to more fully automate our CI/CD processes.

Want to learn more? Register for our upcoming webinar on how to Power up Artifactory with Kustomize & Helm. We’re eager to share what we’ve learned and help you to better manage your deployments!