How can I distribute artifacts from multiple Artifactory instances to target edge nodes?

JFrog Support
2021-02-04 17:51

The following are three methods you can use if you have multiple source Artifactory instances and want to distribute artifacts/files from multiple JPDs (JFrog Platform Deployment)

Distribution scenario: ​​​​​​

  • Geographically dispersed organization developing software around the clock. 
    • One primary JPD in the US East region
    • One JPD in the US West 
    • One JPD in India 
  • All three geographical regions develop and publish builds/artifacts to their respective JPDs.
  • The goal is to deliver the files and metadata from these JPDs to Edges deployed closer to the production instances.

Note: It is recommended to deploy the JFrog Distribution service closer to the source Artifactory instance to avoid latencies and throughput issues. Expect latency if Distribution and source Artifactory are in different JPDs or regions.

Method 1: Promotion to Replication 

Have a central Distribution service installed , preferably in HA mode, in one main JPD. Replicate the required artifacts to be distributed from other JPDs to the main JPD. Now, you can create release bundles in the main JPD and distribute them to the target nodes. Please refer to this document to configure JFrog Distribution

This method is a centralized and efficient way to automate the entire release cycle. You will be consciously pushing qualified files along with the bill of material (BOM) and customer properties, packed and signed into an immutable release bundle, to multiple target nodes securely. This method is based on push distribution, and sometimes consumers may experience 404 errors if a required file doesn’t exist on the target nodes. With this approach, you will also need a separate cleanup mechanism for target nodes.  

User-added imageMethod 2: Smart Repositories 

Using Smart Repositories as Edge level cache. Smart repositories allow you, like Remote Repositories, to cache binaries at an Artifactory Edge node. But, it is more powerful than that. You have full control over ACLs using Access Federation for authorized users to view a source repositories structure on a remote instance of Artifactory. Binaries can then be downloaded based on their type natively. The Smart Repository only caches requests binaries at the Edge. In addition, it provides statistics on downloads back to the source if you want as an option. This provides the ability to extend an Artifactory repository to the Edge while ensuring security and accountability.

This method is for optimized storage on target nodes. Smart repositories are based on pulling required files from the source to the target node on-demand, and consumers will not see any 404s as it’s. You can also sync deletes between source and target nodes. However, the consumer may experience high latency when downloading the files if the network band is low. The files are also not signed and missing the BOM on target nodes. 

User-added image

User-added imageMethod 3 – Multiple Distribution Nodes

Install Distribution service in all JPDs from where you need to distribute artifacts. Register all the Distribution services to the Mission control. You can now create and distribute release bundles from all JDPS and distribute them to the target nodes. Please refer to this document to configure JFrog Distribution.

This method is recommended if you want the segregation of release cycles of your various business units or teams. This decentralized method will let teams manage their release cycles autonomously and prevent a single point of failure from a distribution perspective. However, this method requires deploying distribution services at all development centers and adds an overhead of maintaining them.  

User-added image