Connect your JFrog Artifactory to Docker Hub to Avoid Download Rate Limits

You may have heard the latest Docker announcement about the new rate limits for container image pulls. Starting November 1st, Docker will start to limit Docker Hub usage based on your subscription level and block pull requests that exceed imposed limits. Not only that, Docker has also put in place a new retention policy, six months for free accounts, for inactive images (originally slated for November 1, this policy has been delayed to mid-2021 due to community feedback). These new limitations will make a great impact on how the world’s Docker container images are being consumed.

Docker has set an example as an open source initiative, providing a new level of deployment tools and methodologies for the open-source community. There’s no doubt about that. The new limitations will not make a severe impact on single developers, rather they will bring new challenges for medium to large development teams. The larger your team, the bigger the impact you’ll have. And this may only be the beginning; we may expect to see new policies in the future that affect usage of Docker Hub and beyond.

The good news is that there are many tools available for managing your Docker images to ensure your organization’s development pipeline isn’t affected in any way. JFrog Artifactory is one of the most popular, having functioned as a Docker Registry even before Docker released registry functionality. With Artifactory, you’ll be able to continue managing container images in your own private Docker container registry while reducing the dependency on Docker Hub.

Let’s dive into what these two limitations announced by Docker really mean.

Docker’s new image retention policy

Up until now, Docker images could be stored in Docker Hub for an unlimited period of time. As a result, developers relied on this storage space without caring too much about it. A new image retention policy, in effect in mid-2021, will now be defined according to the new Docker subscription plans. For example, images owned by free accounts will be deleted after 6 months of inactivity.

Docker’s new download throttling

Docker will now set a new limit on data transfer beginning November 1st for free accounts, of 100 pulls for anonymous users and 200 pulls for authenticated/free users, for every 6 hours per IP address or a unique user. A simple calculation of 200 pulls per 6 hours will provide you with approximately 0.55 pulls per minute. This may not be sufficient for you, and reaching the limit will mean that you’re stuck waiting for the next 6 hours time frame. Also, pulling an image that you already have is still counted even if you don’t download the layers. This limitation will prove to be especially challenging for enterprises with a small IP range (sometimes due to corporate VPNs) for their teams on company networks.

Store and Protect your Docker Images in Artifactory

Using Artifactory as your Docker registry allows you to store your Docker images for an unlimited period of time, without worrying about images being expired and deleted. With Artifactory, you’re caching your images and managing registry and retention policies according to what best fits your team (as a best practice, aim to maintain and store only those images that you need on an ongoing basis – this will optimize your Docker hub download rate limits to their fullest). With Artifactory, you also don’t have to worry about your storage exploding. Using checksum-based storage, Artifactory utilizes your storage to its fullest potential.

Reduce the number of pulls to Docker Hub

Also using Artifactory as a remote repository proxying Docker Hub, you’re reducing the number of pull requests you’re making to Docker Hub. Artifactory requests the image you need from Docker Hub once, and makes those images available to all your internal teams using Artifactory without going back to Docker Hub. Artifactory allows you to authenticate with Docker Hub using your Docker Account, so every request will be authenticated and counted based on your account type.

Further, when working with Artifactory versions 6.23 & 7.10 and above and utilizing a proxying remote repository, pull mechanisms from Docker Hub are now efficiently using a new query to better utilize internal caching. This means that before sending a new GET request, (which is considered a pull by Docker, counting against your new restrictions), Artifactory will send a HEAD request to compare manifest files and will update the cached manifest only when needed. Artifactory will pull the image once from Docker, and make the image available across your organization to avoid your pull limits. You can always control the caching rhythm to reduce the calls to Docker hub.

Go Beyond the Docker Registry: Secure and Distribute Docker Images

Now that you have peace of mind knowing your images will always be available and you won’t be throttled or hindered by Docker constraints, you’re ready to tackle the rest of your container’s life cycle.

One of the most important concerns companies have about Docker images is the “Russian doll” problem of multiple containers within containers composed of complex layers that are invisible to standard tools. Thankfully, with JFrog Xray’s baked-in deep recursive scanning of containers stored in Artifactory, layers are exposed and vulnerabilities are identified before you ever move towards production. Beyond just Docker, this security scanning capability is available out of the box for most common package types.

The JFrog Platform also includes multiple tools to distribute software to the edge securely and quickly. With peer-to-peer functionality, JFrog helps you handle download bursts of container images (often several GBs) to hundreds of nodes and clusters. This reduces both latency and the pressure on individual repositories. The JFrog Distribution product also allows you to secure release bundles that may include containers, and deliver them to the edge and validate your software updates – even in air-gapped environments.

The JFrog Platform is uniquely positioned to remediate risk of possible further changes in the Docker offerings, as well as provide you with tools beyond Docker Registry functionality to manage the entire life cycle of your container deployments.

Already a JFrog user? Learn more about using Artifactory as your Docker Registry.

New to JFrog? Take the JFrog Platform for a spin for free in the cloud.