ARTIFACTORY: Troubleshooting Federated repository replication

Gajapathi Kimidi
2022-07-25 13:20

Prerequisites for enabling federated repository replication between Artifactory servers

  1. Artifactory Versions Must be Identical between federated members. 
  2. JFrog Platform Deployments (JPDs) Clocks Must be Synchronized
  3. Before creating federated repositories on Artifactory, it is mandatory to configure custom base url for Artifactory.  In order to configure Custom Base URL Open Artifactory UI > Administration > General > General Settings > Configure Custom Base URL
  4. A circle of trust must be set between the Artifactory instances. There are two ways to establish a circle of trust. 

 

Troubleshooting

1. Base url configuration
When converting a local repository to a federated repository, if you observe the following message
An unexpected error occurred: The Base URL is mandatory and is not configured.”,
you will need to configure the Custom base url first. 

2. Circle of trust
When adding a federated member if you see errors,
"Failed to add federated repository. Make sure that cross-instance authentication is established between this instance and the remote JFrog Platform Deployments.”,
we need to establish circle trust between the federated artifactory members.

3. Full sync of Federated repositories
After converting a local repository to a federated repository, the artifacts that were uploaded before, will not be replicated automatically. We need to run a sync call in order to trigger the replication. curl -u <user> -p <password> -X POST
​​​​​​https://source-artifactory:8082/artifactory/api/federation/fullSync/maven-s1

4. Trailing slash in url
When adding a federated member, make sure to remove any trailing slash(/) at the end of the repository name during the configuration. If there are any slashes at the end of the repository name, we will get the below error message while saving the configuration.Failed to update remote members for federated repository: maven-source-repo, failed members: [http://destination-artifactory:8082/artifactory/maven-dest-repo/]. Please check the URL, network, proxy accessibility, and then try again.

Below is an example of correct configuration. 
 

User-added image
5. Replicated artifacts are visible at the target instance but not downloadable
If you are on a version lower than 7.26.3, the federated replication might fail to replicate the actual contents of the artifacts, and only the database entries will be replicated. That means, we can see the artifacts on the destination side when we expand the Artifactory tree view, however we can not download them. 

Below are the log entries we can see in the destination server at $JFROG_HOME/artifactory/var/log/artifactory-service location. 2022-07-18T09:34:38.584Z [jfrt ] [INFO ] [54af869eb79958bd] [o.a.a.m.i.MirrorImpl:301      ] [http-nio-8081-exec-5] - Mirror member from: http://destination-artifactory:8082/artifactory/maven-d1 to: http://source-artifactory:8082/artifactory/maven-s1 active
2022-07-18T09:34:38.685Z [jfrt ] [INFO ] [70117083326bde1a] [o.a.e.UploadServiceImpl:454   ] [http-nio-8081-exec-3] - Deploy to 'maven-d1:acegi-security/acegi-security/0.5/acegi-security-0.5.jar' Content-Length: 168502 (estimation) artificial: true
2022-07-18T09:34:38.727Z [jfrt ] [ERROR] [5b3a0e48fd06ea0 ] [s.b.c.RemoteBinaryProvider:317] [ated-binary-importer] - Remote endpoint at http://source-artifactory:8082/artifactory/binarystore unreachable for 'mvn-destination-federated-repo-http://source-artifactory:8082/artifactory/mvn-source'. Got response: 503 : Service Unavailable
2022-07-18T09:34:38.736Z [jfrt ] [ERROR] [5b3a0e48fd06ea0 ] [.a.f.b.RemoteBinaryImporter:92] [ated-binary-importer] - Failed to import remote binary with checksum: dc44aac08d4cd9e61047769b52274e8a28a5e268 to local
org.jfrog.storage.binstore.exceptions.BinaryStorageException: 503 : Failed to import binary with sha1: dc44aac08d4cd9e61047769b52274e8a28a5e268
    at org.artifactory.addon.federated.binary.RemoteBinaryImporter.run(RemoteBinaryImporter.java:65)
    at org.artifactory.opentracing.TraceableRunnableDecorator.run(TraceableRunnableDecorator.java:40)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

And when you try to download the file at the destination, you will receive 503 and the below entry will be observed in the artifactory-service.log2022-07-18T09:37:58.665Z [jfrt ] [WARN ] [176c6034415f3e29] [.r.ArtifactoryResponseBase:136] [http-nio-8081-exec-7] - Sending HTTP error code 500: Could not process download request: Binary provider has no content for 'dc44aac08d4cd9e61047769b52274e8a28a5e268'

It is identified as a bug and recommended to upgrade to a version 7.26.3 or above. 

6. GEO based url configuration for federated repositories. 
If you have several Artifactory instances that are connected to a single Geo basedurl

For example you have http://my-artifactory/ where there are two underlying artifactories in two different locations like http://my-EAST-artifactory/ and http://my-WEST-artifactory/ and depending on the user location, the routing happens. In such cases, along with the base url , we need to add the following XML Tag under the <systemProperties/> section in the Global Configuration Descriptor at Administration > Artifactory > Advanced > Config Descriptor. 
Note : We should be careful while editing this file as any unexpected changes to this configuration can corrupt Artifactory. 
For example, for the EAST site, we need the below tag. <federatedRepoUrlBase>https://my-EAST-artifactory/artifactory</federatedRepoUrlBase>

Note: There are a lot of improvements and features added in latest versions of Artifactory with regards to federation. We recommend upgrading to the latest version in order to get all the features of federated repositories.