Introduction
Starting with Artifactory 7.18.3, users can create Federated Repositories, enabling near-instantaneous mirroring of repository content across multiple Artifactory instances. This configuration provides an alternative to traditional Repository Replication, offering potential advantages depending on business needs.
Federated Repositories Task Queueing
Federated Repositories handle artifact mirroring using the following workflow:
1. Artifact Upload: When an artifact is uploaded to any Federation Member, the system initiates the mirroring process.
2. Notification: All other Federation Members receive a notification about the new artifact and the need to import it.
3. Task Creation: Each Federation Member generates an entry in the binaries_tasks table in its database for every artifact requiring import.
4. Import Attempt: Target Artifactory instances process the binaries_tasks table, attempting to fetch the binary from other Federation Members.
5. Completion: Once successfully imported, the respective binaries_tasks entry is removed from the table.
Example Entry in the binaries_tasks Table
| id | repoKey | sha1 | count | created_time | |----|--------------------|------|-------|----------------------| | 1 | sample-repo | abc1 | 2 | 2024-03-01 10:00:00 |
Effects of Network Interruptions on Binary Federation
During step 4 of the Binary Federation process, Artifactory processes the binaries_tasks table in descending order by creation date (oldest tasks first). If network interruptions occur during this process:
- The import attempt fails, and the count field for the task increments.
- The system retries the import until:
A) The binary is successfully imported.
B) The count reaches the maximum retry limit (Default: 10).
If the retry count exceeds the limit, the binaries_tasks entry remains in the database indefinitely. This can lead to a backlog of stale tasks, impacting future federation operations.
Identifying Failed Binary Tasks
Use the Get List of Failed Binary Tasks REST API to detect failed tasks where the count has reached the maximum value.
Recovering Failed Binary Tasks
Stale tasks can disrupt the Binary Federation process. After resolving the underlying network issues, the following steps can help recover affected tasks.
Use the Replay Failed Binary Tasks REST API to mark failed tasks for reprocessing. This can be applied to:
- The entire repository
- A specific binary task within the repository
Handling Non-Existent Binaries
If stale tasks persist even after resetting the count, it may indicate attempted imports of missing binaries. To diagnose this:
1. Check Artifactory Logs
Inspect $JFROG_HOME/artifactory/var/log/artifactory-service.log for errors similar to:
2023-04-12T21:11:27.056Z [jfrt ] [ERROR] - Failed to import binary with sha1: 37e42b3ac0b92eb44dbad92131cdf5eb40d4ca51, repoKey: test-repository, remoteUrl: https://<ARTIFACTORY_URL>/artifactory
2. Verify Binary Existence
Using the sha1 value from the logs, check the filestore on the remote Artifactory instance to confirm the binary’s presence.
3. Purge Non-Existent Tasks
If multiple failed imports correspond to missing binaries, delete them:
For Artifactory 7.68.11 and Above
Use the Delete Failed Binary Tasks REST API to remove failed entries.
Recommendations for Artifactory 7.68.10 and Below
Strong Recommendation: It is highly advised to upgrade to Artifactory's latest version, as numerous Federated Repository improvements have been introduced since version 7.68.10, significantly enhancing reliability and performance.
Identifying Failed Binary Tasks (Artifactory 7.68.10 and Below)
Run the following SQL query directly against the Artifactory database to identify failed tasks:
SELECT * FROM binaries_tasks WHERE count = 10 AND local_repo_key = '<your_repo_key>';
Replaying Failed Binary Tasks (Artifactory 7.68.10 and Below)
Important: Before performing any manual database actions, it is highly recommended to create a database backup. This ensures that if any complications arise during the process, there is a restore point available.
Run the following SQL update query to reset the count and allow reattempts:
UPDATE binaries_tasks SET count = 0 WHERE count = 10;
Resetting the count does not resolve cases where binaries no longer exist on the source Federation Member.
Deleting Non-Existent Binary Tasks (Artifactory 7.68.10 and Below)
Run the following SQL delete query to remove failed tasks:
DELETE FROM binaries_tasks WHERE time >= 1643673600000 AND count > 0;
After deleting failed tasks, initiate a Federated Repository Full Sync to ensure all Federation Members are up-to-date.
Federated Binary Sync Tuning
Artifactory provides various tuning parameters to optimize Federated Repository performance. We recommend reviewing the Federated Repository Tuning Guide for best practices tailored to your environment.