How to Push Replicate everything from one Artifactory to another using the JFrog CLI

Patrick Russell
2021-12-28 22:21

How to Push Replicate everything from one Artifactory to another using the JFrog CLI

Relevant Versions: Artifactory 7.X

Back in Artifactory 6.X, the accompanying Mission Control 3.X was able to set up a Disaster Recovery system. This system involved two separate Artifactory instances kept in sync by Push Replication. The replications would keep the DR Target (PreProduction) updated with the DR Source's production data.

This old system was unfortunately difficult to set up using Mission Control. This setup involved multiple steps. If a step went wrong, Mission Control would not continue or work past the issue. Because of this, JFrog began working on a new DR model internally. 

The core components of the system still exist and can still be used by following the steps outlined in this guide. This is also handy if you want to migrate Artifactory with minimal downtime .

To get the data from one Artifactory to another, Push Replications can be used. This way data is dynamically sent every day to the DR Artifactory.

The only challenge is setting up (potentially) thousands of Push Replication settings. This can be done with some clever scripting and the Artifactory REST API. You'll also need to download the latest JFrog CLI to take advantage of its replication configuration system.

 

Limitations of this Guide

Since this guide sets up Push Replications, the replications will only go one way. Bi-Directional Push Replications will cause serious problems, such as unintended deletions on the Source Artifactory's data. As such, this guide will only set up an Active / Passive Artifactory pair, an Active / Active cluster can be achieved via High Availability or other measures.
 

Create the Repositories and Settings

It's possible to import the Artifactory configurations from Production to Pre-Production using an empty System Export even at scale. At large scales (Above 1 million artifacts) the default Full System Export takes too long to complete. For our purposes, we just need to create the repositories on the other side. 

You can exclude everything but the configurations and users by checking both "Exclude" checkboxes:


 

Note: Due to an open bug, RTFACT-25899, you have to decrypt the passwords before the export in Artifactory 7.X

   curl -u admin http://localhost:8081/artifactory/api/system/decrypt -XPOST

   [Expected Output]
   DONE

This will save a nearly-empty backup of the Production Artifactory's configurations and permissions; it should complete fairly quickly. 

Next, copy the export into the DR Artifactory. Import it using the Import System menu to automatically create all the target repositories:

If both checkboxes are checked on the Import, nothing will be deleted. This way the import can be done again later to update the repositories list and permissions.
 

Replication Setup Step 1: Get the Repository Names

It would take a lot of work to set up Push Replications manually. For many Artifactory users, the act of going through the hundreds, perhaps thousands, of replication configurations would be impossible. However, using new tools added to the JFrog CLI, it's possible to automate this task completely.

It's suggested you create a project folder to save the template files we're about to create:

mkdir replication-setup; cd replication-setup

First, get the repositories list from the Source Artifactory. The command below saves only the repository key, which is the repository's name. The keys are all that we need to set up replications. This command will save a "repositories.list" file we'll use in the next step:
 

# The curl command downloads the list, the grep pulls out the "key" we need #

curl -s -u admin "http://jfrog.7x.gcp/artifactory/api/repositories?type=local" | grep "key" > repositories.list

[Expected output syntax]

With the "repositories.list" file, we've actually gotten all of the information we need to set up replications everywhere. The next step is to create the right template files for the CLI.
 

Replication Setup Step 2: Creating the Replication Settings

The JFrog CLI offers a "replication-create" action , which requires a Replication Template file. In BASH, you can create these template files by using the repositories.list file and some clever processing tricks.

A] First, configure the JFrog CLI with both "source" and "target" Artifactories:
 

jfrog config add source #Define the DR Source Artifactory
jfrog config add target #Define the DR Target

B] Generate the Replication template configuration files. This while loop will read each line of the "repositories.list" file and create a template file based on the key:
 

# The JFrog CLI uses the name "target" and fills in the right URL when replication is configured

cat repositories.list |  while read line 
do   
   #Variable setup
   #Get the repository key, remove "key": from the JSON
   REPO=$(echo $line | cut -d ':' -f 2) 
   REPO_FILENAME=$(echo ${REPO%??} | cut -c 2-) #Get a good looking filename

   #Insert the static default parameters
   echo '{ "enabled": "true","cronExp":"0 0 12 * * ?",' > $REPO_FILENAME-template.json

   #Insert the repository Key
   echo '"repoKey": '$REPO >> $REPO_FILENAME-template.json
 
  #Insert the remaining parameters, note we're replicating to the same repository name
   echo '"serverId": "target", "targetRepoKey": '$REPO' "enableEventReplication":"true" }' >> $REPO_FILENAME-template.json

done

[Expected Output Syntax]

After the loop finishes, please check the output json files. They should all look like the above example image! It's important to double check the templates before we mass-deploy them to the Source to prevent misconfigured settings from being deployed in Production. 

C] Run the "replication-create " CLI command on each of the JSON files, this script will do this for you:

jfrog config use source

ls  | while read line
do
     echo "jfrog rt replication-create $line"
     jfrog rt replication-create $line 
done

This will push each template to the Source Artifactory, and at the end all Artifactory's Local Repositories will be configured to push to the Target:


 

Replication Setup Step 3: Sit back and relax

Believe it or not, that's it! If the previous steps were completed successfully, over time the Production Artifactory will push all of its data to the Target and the two will have the same information. You can kick off the migration by clicking the "Run Now" button from the UI.

Enterprise Plus customers can actively synchronize security information by setting up Access Federation between the two instances.

 

Troubleshooting Issues

If the replication information used in the above steps is incorrect, the Source Artifactory will end up with a lot of bad replication settings. The fastest way to restore the application is to wipe the replication settings in the config descriptor.

The recommended method is to use the REST API to redeploy the "artifactory.config.xml" you have saved in the empty System Export at the start of this process. 

The file to use is located in the export folder, for example 20211215.195625/artifactory.config.xml.​
 

curl -u admin -X POST -H "Content-type:application/xml" --data-binary @artifactory.config.xml http://localhost:8081/artifactory/api/system/configuration

​​​​​​Another faster, but riskier, approach is to use the Artifactory UI Page:

1] Locate the "<localReplications>" block, near the bottom of the file:

2] Select the "<localReplication>" text and scroll down, highlight them all:

3] Delete the entire block and enclose the "<localReplications>" tag:

4] Click "Save", the save checks the syntax and you can reset it if the save fails