ARTIFACTORY: Replication when using the checkBinaryExistenceInFilestore

Scott Mosher
2023-01-22 11:07

To preface this article, this feature does require an Enterprise+ license.  Our wiki outlines the feature and can be found at the below link:

https://www.jfrog.com/confluence/display/JFROG/Repository+Replication#RepositoryReplication-OptimizingRepositoryReplicationUsingStorageLevelSynchronizationOptions

For larger instances with hundreds of terabytes of data, the built-in replication feature within Artifactory will end up taking at minimum months to complete depending on the network.  This is where the feature to check the existence of the binary being replicated comes into play.  With this feature, we rely on transferring the binaries outside of Artifactory from the source instance to the target filestore.  It will need to be stored in the expected checksum format.

Once the binaries are synced, we can then enable replication with the REST API.

https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-SetRepositoryReplicationConfiguration

We will enable the checkBinaryExistenceInFilestore on this replication configuration and now when replication is enabled, the binary will already exist and just the metadata will replicate.  This will end up being much quicker.  The target instance will now receive the replication request and check if the binary is found within the filestore.  If it is found, the target instance will then download the binary, calculate its checksum once more to ensure integrity, and then replicate the metadata.  This is important to know since all the data we have transferred previously will need to be downloaded once more on the target.