How to quickly sync large repositories

Patrick Russell
2019-01-21 23:59

Subject 

How to perform the Content-Free Repository Import to quickly synchronize repositories.

Description

Usually a repository export of a large Artifactory repository is too slow to do in production. However, there is a way to speed up the export-import process for a single repository. After following this procedure, event-based replication can handle new items created on the source Artifactory.

The process involves copying the entire Artifactory filestore to the target machine, then importing the repository metadata. After all the repositories have been updated this way, running the regular Artifactory Garbage Collection routine will scrub the filestore of any unimported data.

Resolution

1. rsync or otherwise copy the source Artifactory’s “filestore” directory to the target Artifactory filestore. Disable Garbage Collection on the target Artifactory to prevent interference with the new filestore.

2. Perform a content-free System Export on the source Artifactory
    Do this by going to Admin -> Import & Export -> System and check the "Exclude Content" checkbox.

3. Perform a Repository Import on the target Artifactory using the source Artifactory export

–   The path for the import should be within the System Export folder: 20180801.194225/repositories/libs-snapshot-local

–   The System export performs the same method as a regular repository import, but captures other data such as security settings that are not needed here

4. Repeat step 3 for any other repositories that need to be fully synced

5. Run Artifactory's Garbage Collection routine to remove the remaining unwanted files from the target filestore

Note: You may observe import errors related to missing binary files, this is from recently uploaded files that the rsync has not captured. These errors will not affect the import of other binary metadata and can be ignored.

The Repository Import routine is only additive, unlike a System Import; any files on the target Artifactory that are not in the export will not be affected.