ARTIFACTORY: Federated Repository Synchronization Tuning

This document provides guidelines and best practices for optimizing the synchronization of federated repositories in Artifactory.

Optimizing the synchronization of Federated Repositories involves adjustments to all the members instances, applicable to all nodes in an HA cluster.

As the federation would add some load to Artifactory, it’s recommended to start by tuning the Artifactory instance as indicated in the following Knowledge Base article.

Federated Repository Configuration Defaults and Tuning

Under this section, we will go over the System Properties related to Federated Repositories which can be adjusted by editing the $JFROG_HOME/artifactory/var/etc/artifactory/artifactory.system.properties file and restarting Artifactory:

Configuration Values

Maximum Configuration Threads:
- Property: artifactory.federated.max.config.threads=5
- Default Value: 5
- Description: This setting specifies the maximum number of threads available for configuration operations in federated repositories. It differs from the "numberOfRemoteImporters" property in binarystore.xml, which controls remote pulls. Instead, this setting manages the propagation of members' repository configuration changes.

Maximum Total HTTP Connections:
- Property: artifactory.federated.repo.max.total.http.connections=50
- Default Value: 50
- Description: This setting controls how many HTTP connections can be opened per Federation sync job, while the number of sync jobs is determined by the Maximum Threads Percentage / Maximum Threads Count mentioned below (depending on the version).

Maximum Threads Percentage (Artifactory versions below 7.71):
- Property: artifactory.federated.repo.max.threads.percent=10
- Default Value: 10
- Description: This increases the total allocation of the Artifactory Task Queue to Federation events. The Task Queue also controls background tasks such as Metadata calculations, binaries cleanup, and Push Replications.

Maximum Threads Count (Artifactory versions starting from 7.71):
- Property: artifactory.federated.work.queue.maxPoolSize
- Default Value: 12
- Description: This property sets the maximum number of threads available for the work queue in Federated repository tasks. It can be increased to optimize the throughput of synchronization activities between federated instances, ensuring efficient processing of tasks such as repository synchronization and metadata handling.

💡 Note: The System Property artifactory.federated.repo.max.threads.percent was replaced starting from version 7.71 to artifactory.federated.work.queue.maxPoolSize. Any users who employ the previous property must change to the new one as part of this upgrade. In addition, the default value of this property has been changed to 12 threads. This was done to align the default thread values on the source and target instances of the queue mechanism to avoid potential bottlenecks.

Executor Pool Maximum Queue Size:
- Property: artifactory.federated.repo.executor.poolMaxQueueSize=10000
- Description: This setting controls the overall size of the Federated Sync Queue. As we are increasing the thread we may see a higher queue so we can increase this number accordingly.
- Default Value: 10000

Suggestions for Tuning

For enhanced performance and efficiency in handling federated repository events, you might consider adjusting these values:

Configuration Threads:
- Increase artifactory.federated.max.config.threads to 10 to enhance the processing of changes in repository configurations. This is relevant for propagating changes across member repositories in large setups where the number of members in a federated repository exceeds the default value of 5. This ensures that updates to all members are made in parallel without delays.
Total HTTP Connections:
- Increase to artifactory.federated.repo.max.total.http.connections=100 to allow more concurrent connections for sync.
Threads Percentage (Artifactory versions below 7.71):
- Raise to artifactory.federated.repo.max.threads.percent=15 to allocate a higher proportion of Artifactory's task queue to federation events.
Maximum Threads Count (Artifactory versions starting from 7.71):
- Raise to artifactory.federated.work.queue.maxPoolSize=18 to allocate a higher amount of threads of Artifactory's task queue to federation events.
Executor Pool Queue Size:
- Expand to artifactory.federated.repo.executor.poolMaxQueueSize=30000 to accommodate a larger queue of pending tasks, minimizing potential bottlenecks during peak loads.

Default values:

artifactory.federated.max.config.threads=5
artifactory.federated.repo.max.total.http.connections=50
artifactory.federated.work.queue.maxPoolSize=12
artifactory.federated.repo.executor.poolMaxQueueSize=10000

Tuning:
To further optimize, consider increasing these values and closely monitor both the federation and instance performance:

artifactory.federated.max.config.threads=10
artifactory.federated.repo.max.total.http.connections=100
artifactory.federated.work.queue.maxPoolSize=18
artifactory.federated.repo.executor.poolMaxQueueSize=30000

Filestore Tuning

While the above controls the tuning of the artifacts, tuning the Filestore Configuration can also enhance the synchronization of binaries.

In this section, we will cover the federated repository configuration, which defines the filestore settings located under $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml (followed by a restart) for synchronizing federated repository binaries.

Configuration Parameters:

numberOfRemoteImporters: This setting defines the number of workers responsible for downloading binaries from remote federated binary providers. It defines the parallelism for pulling binaries from remote sources once metadata has been synced as a binary task. (Note: While the source instance pushes metadata for artifacts to the target instance, the corresponding binary, however, is pulled on the target instance from the source instance's federated binary provider). The default setting is 6.
numberOfLocalImporters: numberOfLocalImporters: Specifies the number of workers dedicated to on-demand downloads. In this scenario, binaries are pulled by the target repository when requested, even if their binary has not been previously fetched. This mechanism ensures that binaries are available as soon as their metadata is available, prioritizing them over binary tasks queued for sync by the remote importers. This helps prevent delays in artifact availability when users request them. The default setting is 6.
maxRetry: Determines the number of attempts to retry import of a binary before it is considered as a fetch failure. Increasing this value allows for additional attempts to process failed binaries. The default is set to 10.
firstRetryDelayMs: The initial delay before retrying a binary import, in milliseconds. The default delay is 30,000 ms (30 seconds).
maximumIdleTimeMs: The maximum wait time between task assignments. The default is 60,000 ms (1 minute).
maximumExecTimeMs: The maximum duration a task can be held by a single node. The default is set to 3,600,000 ms (60 minutes).

If these parameters are not explicitly set in your binarystore.xml , the "federated-repo" configuration will default to the following values:

<?xml version="1.0" encoding="UTF-8"?>
<config version="1">
    <chain template="file-system"/>
    <provider id="federated-repo" type="federated-repo">
        <numberOfRemoteImporters>6</numberOfRemoteImporters>
        <numberOfLocalImporters>6</numberOfLocalImporters>
        <firstRetryDelayMs>30000</firstRetryDelayMs>
        <maxRetry>10</maxRetry>
        <maximumIdleTimeMs>60000</maximumIdleTimeMs>
        <maximumExecTimeMs>3600000</maximumExecTimeMs>
    </provider>
</config>

Example Tuning:

To tune these settings, you can modify the "federated-repo" provider section in your $JFROG_HOME/var/etc/artifactory/binarystore.xml file as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<config version="1">
    <chain template="file-system"/>
    <provider id="federated-repo" type="federated-repo">
        <numberOfRemoteImporters>12</numberOfRemoteImporters>
        <numberOfLocalImporters>12</numberOfLocalImporters>
        <firstRetryDelayMs>30000</firstRetryDelayMs>
        <maxRetry>15</maxRetry>
        <maximumIdleTimeMs>60000</maximumIdleTimeMs>
        <maximumExecTimeMs>3600000</maximumExecTimeMs>
    </provider>
</config>

In this example we increased the number of remote and local importers, as well as the maximum retry limit, to enhance the performance of sync process and resilience of the failed federated repository synchronization tasks.

Federation Recovery and Auto-Healing:

In certain cases, maintaining near real-time synchronization of all artifact events (create, update, delete) among Federation members may not be possible to maintain near real-time synchronization of all artifact events due to:
- Short-term networking issues between the JPDs
- Artifactory upgrades
- User-initiated synchronization pause
- Other similar scenarios.
If synchronization continues to fail after reaching the maximum number of retry events, event sync is paused, and the Federation transitions into an error state.

To recover the Federation, one approach is to perform a full sync. However, this can be time-consuming, especially if the Federated repositories contain a large number of artifacts, effectively restarting the Federation.

Starting with release 7.71.1, Artifactory introduces an auto-healing mechanism. This mechanism regularly checks Federated repositories for exhausted queues (queues that have exceeded the maximum number of attempts to send events to other Federation members). If such queues are detected, the mechanism resets the failed events automatically and attempts to sync with the target mirror again. See Federation Recovery and Auto-Healing for more information.

Enabling Auto Healing in Artifactory Versions Below 7.82

Overview
For Artifactory versions below 7.82 (above 7.71.1), it’s necessary to enable the artifactory.auto.healing.enabled=true property to enable the Auto Healing feature. Note that starting from version 7.82, Auto Healing is always enabled.

💡 We recommend that any customer on version 7.71.1 and above enable the Auto Healing feature.

To enable Auto Healing, add the following property to your Artifactory System Properties file: artifactory.auto.healing.enabled=true
To verify that auto-healing is enabled in your environment, you can double-check by ensuring that the property is set to true. You can do this effectively by running the System REST API and grepping for the property name: “artifactory.auto.healing.enabled” as shown below:

curl -u<ADMIN_USER> <http://<JFROG_URL>/artifactory/api/system> | grep artifactory.auto.healing.enabled

It's important to note that this feature requires all JPDs in the Federation to run Artifactory release 7.71.1 or later.