Artifactory as a Caching Mechanism for Package Managers

As a developer in the JFrog Ecosystem team, I rely on many package managers to download tools for my builds. Pre-build tools such as Curl, CLI, wget, Maven, Gradle, npm and others, used to build the final deliverable, are automatically downloaded by the CI server as part of the first step of our builds. Relying on external servers for downloading these pre-build tools, can sometimes cause stability issues, also called “Environmental Issues”, that will break the build. This advanced technical blog post will show how to overcome these challenges using Artifactory remote repositories. It will be the first of a 2-part series.

The Pain: Package managers’ internal repository is not always stable

We have a variety of package managers to choose from. Chocolatey for Windows, Homebrew for Mac/Linux, npm, Yum, APT, Docker, Flatpak, Snap, and the list goes on. Sometimes when trying to download the tools we need from these package managers, we may get unexpected 5xx server errors. Moreover, downloading packages from those package managers can be slow. This problem becomes 1000x worse when using automatic build tools (such as Travis CI, Jenkins, Appveyor, etc). The results are slow and unstable builds. One way to deal with these environmental issues is by manually running the build again. However, this doesn’t solve the problem. Also, in some cases it’s not even possible to download these external tools because of  internal network policies.

The Solution: Use Artifactory remote repositories

An Artifactory remote repository serves as a proxy to a repository located on a remote server. Requests to this Artifactory remote repository will first try to resolve the artifact from the local cache, and if the artifact cannot be found locally, Artifactory will attempt to resolve the artifact from the remote server and then continue to store that artifact in the local cache for future requests.

Using remote repositories as a proxy to download packages from package managers can make your builds faster and more stable. They also provide an isolated and secure environment for your build, especially if your organization uses an internal network as mentioned above.

Let’s see this in action, with an example using the Chocolatey package manager.

Important Notice: For your security, make sure to mask passwords from the console output.

Use Case: Chocolatey

Issue: 503 Error

Our JFrog CLI tests use Choco to download Gradle for Windows agents. We often noticed that some of the builds failed due to the following 503 error:

Failures
 - gradle (exited 1) - gradle not installed. An error occurred during installation:
 The remote server returned an error: (503) Server Unavailable. Service Unavailable

Resolution

After struggling with this challenge, the solution we found was actually quite simple. Define a remote Nuget repository in our own Artifactory instance and use it as a source to download packages from Choco package manager.

Step by step guide

Step 1: Configuring Artifactory repository

Create a remote Nuget repository in Artifactory:

• Repository Key: choco
• Url: https://chocolatey.org

Step 2: Install Choco package

Using anonymous install commands

choco install <package-name> -s <artifactory-url>api/nuget/choco

Using authenticated access

choco install <package-name> -s <artifactory-url>api/nuget/choco -u <artifactory-user> -p <artifactory-password>

In my next blog post I’ll continue with additional use cases for other tools, including Homebrew and Yum. Try these out for yourself!