Enable Multi-Site DevOps with Federated Repositories

The days when applications were created by a small team of developers in one room are long past. Enterprise software development is now a highly collaborative endeavour of packages shared by intersecting teams across multiple sites spread across the globe.

For the enterprise, JFrog Artifactory has long enabled multi-site replication through different push/pull replication topology options, empowering geographically distributed teams to work on the same artifacts (binaries and their metadata), with minimal latency through local repositories.

Most recently, we’ve introduced JFrog Artifactory Federated Repositories, an innovative, bidirectional mirroring technology that provides DevOps teams an option that is easy to set up and maintain for multi-site teams and projects. This repository mirroring technology continuously synchronizes a federated set of repositories across multiple sites.

With JFrog Artifactory federated repositories, enterprises can:

• Collaboratively develop software across geographically distributed teams
• Synchronize  frequently updated binaries between sites securely and efficiently
• Scale development globally
• Ensure all shared artifacts and metadata are current to the latest version everywhere
• Replicate for disaster recovery

Scaling Globally with Federated Repositories

A JFrog federated repository enables multi-site development by empowering geographically distributed teams to collectively share artifacts and their metadata. 

Through federation, a local repository in one Artifactory deployment (for example, in Amsterdam) can be logically joined to synchronize with a local repository in another Artifactory deployment (for example, in Bangkok). When joined in this way, artifacts and metadata pushed to the repository in Amsterdam will automatically be replicated to the repository in Bangkok. And because this federation is bidirectional, data pushed to the repository in Bangkok is replicated in Amsterdam as well.

In this way, Artifactory repositories joined in a federation provide each of these sites a unified, locally accessible repository of shared global data.

Multi-Site Synchronization

Easy Administration

With a JFrog federated repository, an authorized administrator at one site can create and join local repositories at several JFrog Platform deployments into a single, multi-site federation. There’s no need to coordinate setup between platform administrators at different physical sites across time zones to create your federation topology. 

Setup and maintenance can be quickly and easily accomplished from any affiliated physical site: all changes to configuration and repository settings are synchronized automatically across all federation members.

Administrator View

Scalable for the Enterprise

You can federate repositories among any of the 30 repository types supported in Artifactory’s universal package management. All repositories in a federation must be of the same type, of course, but you can create multiple federations for different package types. For example, you can have a federation of npm repositories, another of Maven repos, and another for Docker.

Any repository federation can include up to 10 members (local repositories in different JFrog Platform deployments (JPDs) at remote sites) of the same repository type, providing broad geographical coverage. 

Secure Circle of Trust

Federated repositories use binary provider tokens to establish a circle of trust among members without having to set up certificates at each site. JFrog Mission Control automatically enables all JPDs for secure federation, or you can manually specify secure JFrog Platform URLs.

Administrators at each site can enable or restrict access to federated repositories by their own users through the permission groups managed on their own JPD.

How to Configure Federated Repositories

To start working with Federated repositories, you will first need to set up cross-instance authentication (Circle-of-Trust) between the JPDs. Once this is configured, the platform administrator may perform CRUD (Create, Remove, Update and Delete) actions on the repositories based on their predefined set of permissions. 

All the actions performed on one of the Federated repositories will automatically be synchronized and reflected on all of the Federated members, including not just the repository content, but also key configuration changes.

It’s important to understand that, within a federation, bidirectional mirroring of artifacts and metadata only occurs between repositories that have declared a direct connection between them. Mirroring does not automatically cascade to other, secondarily connected repositories.

For example, consider an multi-site example with these four JFrog Platform deployments:

1. A cloud instance on Amazon Web Services (AWS)
2. An on-prem site at the main data center in Orlando, Florida
3. An on-prem site at corporate headquarters in Santa Clara, California
4. An on-prem site at a branch development office in Waltham, Massachusetts

Let’s look at some example topologies to see how this works.

Star Topology

In a star topology, a local repository in a JPD acts as a central hub to other JPD repositories in the federation. The hub JPD repository shares its artifacts and metata with everyone in the federation, and everyone shares theirs with the hub. However, the other sites do not share with each other.

In this example, we configure a federation from the cloud instance on AWS as the central hub to connect to the affiliate sites. No federated repository configuration takes place at the three on-prem sites.

So in this example:

• An artifact pushed to the repository in AWS will be mirrored in Orlando, Santa Clara, and Waltham
• An artifact pushed to the repository in Orlando will be mirrored only on AWS.
• An artifact pushed to the repository in Santa Clara will be mirrored only on AWS
• An artifact pushed to the repository in Waltham will be mirrored only on AWS

	Orlando	Waltham	Santa Clara	AWS
Orlando	✅	X	X	✅
Waltham	X	✅	X	✅
Santa Clara	X	X	✅	✅
AWS	✅	✅	✅	✅

 

As can be seen, there are significant limitations to this topology. For instance, artifacts created in Orlando will not be available locally to developers in Waltham (although all sites can access all artifacts through the cloud instance on AWS).

This type of topology may be useful in certain situations, such as when most artifacts and metadata are expected to be produced at a central site, but need to be available locally to affiliate sites.

Full Mesh Topology

In a full mesh topology, all artifacts and metadata in the federation are mirrored in local repositories at all sites. All users at all sites have access to all data according to their locally administered permissions, with minimal network latency.

To accomplish this, platform administrators must be careful to be complete in their federation connections. 

For instance, the admins in our example organization might mistakenly configure a federation as shown in this diagram:

While this topology diagram may appear to create a full mesh network through this circular set of connections, it does not. Bidirectional mirroring of artifacts and metadata only occurs between repositories that are directly connected in the federation.

In this configuration, each site will mirror only two of the others, not all three:

	Orlando	Waltham	Santa Clara	AWS
Orlando	✅	✅	X	✅
Waltham	✅	✅	✅	X
Santa Clara	X	✅	✅	✅
AWS	✅	X	✅	✅

 

To correctly configure a federated repository for full mesh, each site must be configured to directly federate with the other three.

In this full mesh configuration, each site will mirror all others:

	Orlando	Waltham	Santa Clara	AWS
Orlando	✅	✅	✅	✅
Waltham	✅	✅	✅	✅
Santa Clara	✅	✅	✅	✅
AWS	✅	✅	✅	✅

 

From Many, One

Federated repositories offer a flexible way to share artifact repositories across multiple sites that’s easy to set up and maintain. How you configure your topology is up to you, in whatever style best fits your needs.

Once configured, a federated repository acts as a single unit, addressable by developers at each site as a single, local URL. Because artifacts and their metadata are mirrored to all repositories, each site can function at hyper-local speeds, protected from intermittent or severed connections.

Security can still be managed at each site, with local administrators governing individual or team repository access through the JFrog Platform’s fine-grained permissions controls.

Want to learn more? You can request a free trial of the JFrog Platform (Enterprise or Enterprise+) to explore for yourself!