5 Best Practices for GoLang CI/CD

For developers programming in long-established languages like Java, JavaScript or Python, the best way to build continuous integration and continuous delivery (CI/CD) workflows with Artifactory is pretty familiar. A mature set of dependency management systems for those languages and container solutions like Docker provide a clear roadmap.

But if you’re programming your applications in GoLang, how hard is it to practice CI/CD with the same kind of efficiency?

As it turns out, it’s gotten a lot easier, especially with some of the latest innovations in Go. With Artifactory’s native support for GoLang, the path to quality CI/CD is much clearer.

Best Practices

At JFrog, we’re big fans of GoLang, using it as the language for several of our flagship solutions. And we practice what we promote, too, using Artifactory at the heart of our CI/CD. Here are some of the practices we can recommend:

1. Use Go Modules

Unlike many established programming languages, initial releases of GoLang didn’t provide a common mechanism for managing versioned dependencies. Instead, the Go team encouraged others to develop add-on tools for Go package versioning.

That changed with the release of Go 1.11 in August, 2018, with support for Go modules. Now the native dependency management solution for GoLang, Go modules are collections of related Go packages that are versioned together as a single unit. This enables developers to share code without repeatedly downloading it. 

A Go module is defined by a go.mod file in the project’s root directory, which specifies the module name along with its module dependencies. The module dependency is represented by module name and version number.

If you haven’t adopted Go modules yet, then you’ll need to follow the steps below:

1. Use go mod init to generate a go.mod file if previous package managers were used.
2. Use go mod tidy if other package managers were not used. This command will generate a populated go.mod file.
3. If the version is v2 and above, then this requires the module name change to add the corresponding suffix, updates to the import path, usage of module aware static analysis tools and finally update code generator files such as .proto files to reflect the new import path.

For example, here is a go.mod file for a publicly available Go module for a structured logger:

module github.com/sirupsen/logrus

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/konsorten/go-windows-terminal-sequences v1.0.1
	github.com/pmezard/go-difflib v1.0.0 // indirect
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.2.2
	golang.org/x/sys v0.0.0-20190422165155-953cdadca894
)

Note that the version numbers must conform to semver convention (for example, v1.2.1 instead of 20190812, or 1.2.1) as required by the go command. You should avoid using pseudo versions like the one shown above (v0.0.0-yyyymmddhhmmss-abcdefabcdef) — although commit hash pseudo-version were introduced to bring Go Modules support to untagged projects, they should be only used as a fallback mechanism. If the dependencies you need have release tags, use those tags in your require statements.

In your own code, you can import this Go module along with other dependencies:

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"os"

// Public Go Module for logging
	log "github.com/sirupsen/logrus"
)

Then you can reference the module functions in your GoLang code:

   // Send text to the log
   log.Printf("Hello Log!")

2. Use GOPROXY to Ensure Immutability and Availability

Once you maintain your GoLang dependencies as versioned modules, you can keep them as immutable entities by setting up a GOPROXY. In this way, you can always guarantee what a specific version of a module contains, so your builds are always repeatable.

Will you be using Go modules that are publicly available open-source packages? Ones you create and share with the OSS community? Modules you keep private to just your team? Here are ways to do each, or all at once, and ensure that those unchanging versioned modules are always available for your builds.

GOLANG.ORG

The Go team at Google maintains a set of GOPROXY services for publicly available modules. As of Go 1.13, the module mirror proxy.golang.org is automatically set as the default GOPROXY when Go is installed or updated. This service is also supported by an index service for discovering new modules (index.golang.org), and a global go.sum database for authenticating module content (sum.golang.org).

The go.dev site is also maintained by the Go team and is the hub for Go users providing centralized and curated resources from across the Go ecosystem. Within this site, you can explore pkg.go.dev to search among thousands of open-source Go packages through a friendly UI.

Artifactory Go Registries

While it’s good to share, you’ll likely also need to restrict the use of some Go modules you create to within your organization. With Artifactory, you can set up both local and remote Go Registries, making public and private modules equally available to your builds.

You can proxy the Go mirror service as a remote Go registry in Artifactory, providing your build system with a local cache that further speeds up your builds and helps protect against network connection outages. Setting one up is easy.

You can also configure one or more local Go registries for modules you need to maintain privately within your organization.

When you combine local and remote registries into a Virtual Repository, your builds can resolve Go module dependencies from both a public source and the modules you create and maintain privately.

You can learn more about how to configure your GOPROXY for Artifactory repositories through this blog post on Choosing Your GOPROXY for Go Modules.

3. Use Artifactory Repository Layouts

When you build your app (using go build), where will the binary artifacts generated be stored? Your build process can push those intermediate results to repositories in a binaries repository manager like Artifactory.

For these intermediate Go artifacts, you’ll use Artifactory’s generic repositories. Structuring those repositories in a smart way can help you control the flow of your binaries through development, test, and production with separate repositories for each of those stages. To help you do this, use the Custom Repository Layout feature of Artifactory with those generic repositories.

Create a custom layout that is similar to the following:

[org]/[name<.+>]/[module]-[arch<.+>]-[baseRev].[ext]

When you configure the custom layout, you should test the artifact path resolution to confirm how Artifactory will build module information from the path using the layout definitions.

4. Build Once and Promote

With your repositories properly set up for your Go builds, you can start to move them through your pipeline stages efficiently. 

Many software development procedures require a fresh complete or partial build at each staging transition of development, testing, and production. But as developers continue to change shared code, each new build introduces new uncertainties; you can’t be certain what’s in it. Even with safeguards to help assure deterministic builds, that may still require repeating the same quality checks in each stage.  

Instead, build your Go-based microservices once, then promote them to the next stage once promotion criteria such as tests or scans are met. If you plan to containerize your Go microservice, the same principle applies: build each Docker image once and promote it through a series of staging repositories. In this way, you can guarantee that what was tested is exactly what is being released to production.

5. Avoid Monolithic Pipelines

Instead of a single, monolithic pipeline for your app, It’s better to have several, with each one building, testing, scanning and promoting a different layer. This helps make your CI/CD process more flexible, with different teams responsible for each layer, and fosters a “fail fast” system that helps catch errors early.

For example, deploying a containerized application can typically be composed of five pipelines:

1. Build the Go application using the JFrog CLI. This pipeline pulls the source code; builds the application with Artifactory repositories for dependencies and output binaries; and tests and promotes from a dev repo to a staging repository in Artifactory.
2. Build a base layer of the containerized app, e.g. a Docker framework. Static or dynamic tags can be used based on company’s risk and upgrade policies. For example, If a dynamic tag such as alpine:3.10 is used as a base layer of Docker framework, then all patch updates will be included each time the Docker framework is built. This pipeline will include build, test and promote stages.
3. Pull the promoted artifacts produced by the prior pipelines and build a containerized app.This will also have build, test, scan and promote stages. 
4. Build a Helm chart that points to a statically tagged & promoted version of a containerized app that was produced by prior pipeline.
5. Deploy the containerized Go app to Kubernetes using the Helm chart.

Artifactory acts as your “source of truth” for your Go builds, providing a GOPROXY for both public and private modules, as well as storing compiled binaries. Using the JFrog CLI to build your Go app helps the build process interact with Artifactory, and capture the build info that makes your builds fully traceable. Here is a sample snippet:

// Configure Artifactory
jfrog rt c

// Configure the project's repositories
jfrog rt go-config

// Build the project with go and resolve the project dependencies
from Artifactory.
jfrog rt go build --build-name=my-build --build-number=1

// Publish the package we build to Artifactory.
jfrog rt gp go v1.0.0 --build-name=my-build --build-number=1

// Collect environment variables and add them to the build info.
jfrog rt bce my-build 1

// Publish the build info to Artifactory.
jfrog rt bp my-build 1

To explore this more, take a look at our example demonstration files for GoLang CI/CD  that we’ll be using at GopherCon.

Boldly Go

As you can see, some simple best practices in managing your GoLang applications can ease the way to effective CI/CD. And Artifactory, as the essential manager of your software artifact supply chain, can play a central role in helping to bring those methods into your software development pipeline.

Any questions? We’ll be happy to answer them at Gophercon; dive in with us to explore all the best ways to release your Go applications fast, and at top quality. Or you can start trying to put them into practice with a free trial of Artifactory!