How to mirror a remote repository

JFrog Support
2016-10-06 13:35

Mirroring a remote repository is not recommended for the reasons discussed below. Instead, we suggest caching only the needed artifacts. For example, you may complete the builds using a remote repository, so all needed packages are cached and available for following builds.

If the mirroring is still required, you may mirror the remote repository using recursive HTTP HEAD requests, using wget or curl to spider the remote repository. For example, to pull the whole remote repository to Artifactory "debian-remote" repository, run the following: wget –spider -o ~/wget.log -w 1 -r -np


Please be cautious about using the method for these reasons:

1. Pulling everything would mean you are downloading and storing versions of packages that you may never use.

2. The repository index will include everything, so the searching the index will take longer time

3. Remote repositories of this kind may block users that try to mirror them in an effort to conserve bandwidth.