Go Modules Are Awesome, But There Is One Tiny Problem

Note: this blog post has been updated on October 12, 2021

Most programming languages and tools support the concept of packages and most of the times those packages have versions. The great thing about versions, whether they’re used by tools or not, is that they allow developers to clearly communicate what the dependencies are to build the final product.

Go has struggled quite a bit with versioning ever since its inception and that led to developers starting to build their own tools like gb and glide. In early 2017, the Go team launched Dep, which was the official experiment for package management in Go. The lessons learned from Dep, and other languages, led to the official proposal for Go Modules which got introduced in Go 1.11.

Did you know? that JFrog Xray supports auditing Go projects — scanning your Go packages for security vulnerabilities and license compliance.

So, what’s the problem?

Don’t get me wrong, I’m very excited about the introduction of Go Modules as it solves a bunch of problems. It also moves us away from the ugly concept of vendoring. Let’s be honest, who wants to keep a copy of the sources of every dependency in their project, risk losing dependencies all together (we’ve learned from left-pad, right?) or have their colleagues experiment a little with new versions?

Those are all problems that Go Modules solve, and it even introduces an environment variable called GOPROXY which allows developers to send all module download requests through a proxy. That in itself is obviously great for companies that want to keep a cache of versions (and not download the dependency every single time), keep a tight eye on security (and limit access to the Internet from their build servers), and many other reasons. There’s just one slight problem, though. The GOPROXY is hardly usable.

The problem with GOPROXY

Currently, using GOPROXY is an “all or nothing” exercise. It means that all dependencies in your entire dependency tree (both direct and transitive) need to be resolved from that proxy server and the go client will fail if any of the dependencies are not present in that Go registry.

In other words, if someone builds a new app with labstack/echo v3.3.5 and dgrijalva/jwt-go v3.2.0 in their go.mod file and run go install with the GOPROXY variable set, the go client will fail unless both modules, their requested versions, and all their transitive dependencies, are available on that proxy. So unless I’m absolutely sure that my entire dependency tree can be resolved from the proxy server, which can be internal or external, using this approach will not work.

Fallbacks

As with every problem, there has to be a solution, right? Right! In fact, the solution already exists. A thread in the Golang repository on GitHub (started by Aaron Schlesinger from the Athens team), outlines the problem and the solution to allow proxies to supply only some modules. Looking back at the example earlier, if one of the two modules is available on the proxy and one isn’t, the build will succeed. One module will be downloaded from the proxy and the other will be downloaded from a version control system.

The option to have such a fallback would solve part of the problem. Some Go Module proxies, like Athens, will be able to construct empty go.mod files from those unresolved requests, which in turn causes other issues.

Completing the solution

The ideal solution would allow users to specify their GOPROXY, use fallbacks in case the module isn’t available and, the important thing, upload those modules to the proxy. This approach means only one developer would ever download a version of a module and after that, it will be available in the proxy server for use by other developers and CI servers. As not all companies will allow their CI servers to connect to the internet, this would really solve the problem and make Go Modules flawless for developers. If you’re wondering how that might look, check out the JFrog CLI which has all of this logic built in. I think we all agree though, that this is a feature we expect from the language itself…