ARTIFACTORY: How to resolve JFrog CLI downloading files that failed from the virtual repository that contains multiple smart repositories

AuthorFullName__c
Sun Jinlong
articleNumber
000006314
FirstPublishedDate
2025-01-06T09:49:39Z
lastModifiedDate
2025-08-01

ARTIFACTORY: How to resolve JFrog CLI downloading files that failed from the virtual repository that contains multiple smart repositories

Introduction

When utilizing the JFrog CLI to retrieve files from a virtual repository that encompasses several smart remote repositories, one might experience issues such as file download failures or incomplete downloads. 
Here is the example that may be encountered:
  • Create several local repositories with Generic type on Artifactory node 1 and upload the test files separately.


    User-added image 
  • Create several smart remote repositories on Artifactory node 2 that proxy local repositories on Artifactory node 1 

    User-added image 
  • Create a virtual repository with Generic type named test-generic-virtual on Artifactory node 2 which includes all the above smart remote repositories

    User-added image 
  • When using the JFrog CLI to download files from the virtual repository test-generic-virtual, we may encounter failure or incomplete file downloads.
    • In this example, the test-generic-virtual repository contains seven smart remote repositories, smart remote repository proxy generic local repository at Artifactory node 1.
    • When downloading all the files in the local repository at Artifactory node 1 through the test-generic-virtual, we may encounter an incomplete download.

      User-added image 
    • There are seven files on the Artifactory node 1, but the JFrog CLI only downloads five files.

 

Root Cause

We can analyze the reasons for the above phenomenon from the JFrog CLI logs and Artifactory logs.
  • The JFrog CLI debug log:
    [Debug] Usage Report: Sending info...
[Debug] Sending HTTP GET request to: http://<art_node2>:8082/artifactory/api/system/version
[Debug] Sending HTTP GET request to: http://<art_node2>:8082/artifactory/api/system/version
[Debug] Artifactory response: 200 OK
[Debug] JFrog Artifactory version is: 7.98.11
[Debug] Sending HTTP POST request to: http://<art_node2>:8082/artifactory/api/system/usage
[Debug] Artifactory response: 200 OK
[Debug] JFrog Artifactory version is: 7.98.11
[Debug] Searching Artifactory using AQL query:
 items.find({"$or":[{"$and":[{"repo":"test-generic-virtual","path":{"$match":"*"},"name":{"$match":"*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","sha256","size","type","modified","created","property").transitive()
[Debug] Sending HTTP POST request to: http://<art_node2>:8082/artifactory/api/search/aql
[Debug] Artifactory response: 200 OK
[Debug] Streaming data to file...
[Debug] Finished streaming data successfully.
[Info] [Thread 0] Downloading "test-generic-remote-2/test2.file" to "test2.file"
......
......
[Debug] [Thread 2] Artifactory response: 200 OK
[Debug] [Thread 0] Artifactory response: 200 OK
[Debug] Downloaded 5 artifacts.
     
    • We can get that the JFrog CLI sends an AQL query request to the test-generic-virtual repository before downloading the files
  • Artifactory node 2 Artifactory logs:
      
    # The AQL request only return five repo’s artifacts to download.
artifactory-request.log:2247:2024-12-27T08:31:32.885Z|7c5824a5cf507379|198.19.249.3|admin|POST|/api/search/aql|200|233|2106|203|jfrog-cli-go/2.71.0
artifactory-request.log:2248:2024-12-27T08:31:32.967Z|7c5824a5cf507379|198.19.249.3|admin|GET|/test-generic-remote-3/test3.file|200|-1|2|28|jfrog-cli-go/2.71.0
artifactory-request.log:2249:2024-12-27T08:31:32.972Z|7c5824a5cf507379|198.19.249.3|admin|GET|/test-generic-remote-5/test5.file|200|-1|2|33|jfrog-cli-go/2.71.0
artifactory-request.log:2250:2024-12-27T08:31:32.996Z|7c5824a5cf507379|198.19.249.3|admin|GET|/test-generic-remote-6/test6.file|200|-1|2|25|jfrog-cli-go/2.71.0
artifactory-request.log:2251:2024-12-27T08:31:33.000Z|7c5824a5cf507379|198.19.249.3|admin|GET|/test-generic-remote-7/test7.file|200|-1|2|23|jfrog-cli-go/2.71.0
artifactory-request.log:2252:2024-12-27T08:31:33.002Z|7c5824a5cf507379|198.19.249.3|admin|GET|/test-generic-remote-2/test2.file|200|-1|2|63|jfrog-cli-go/2.71.0

# only Download five Artifacts from proxy repo
artifactory-request-out.log:96:2024-12-27T08:31:32.842Z|7c5824a5cf507379|test-generic-remote-7|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-7/|200|0|0|156
artifactory-request-out.log:97:2024-12-27T08:31:32.847Z|7c5824a5cf507379|test-generic-remote-7|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-7/|200|0|0|4
artifactory-request-out.log:98:2024-12-27T08:31:32.854Z|7c5824a5cf507379|test-generic-remote-6|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-6/|200|0|0|6
artifactory-request-out.log:99:2024-12-27T08:31:32.861Z|7c5824a5cf507379|test-generic-remote-3|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-3/|200|0|0|7
artifactory-request-out.log:100:2024-12-27T08:31:32.868Z|7c5824a5cf507379|test-generic-remote-5|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-5/|200|0|0|7
artifactory-request-out.log:101:2024-12-27T08:31:32.878Z|7c5824a5cf507379|test-generic-remote-2|admin|POST|http://<art_node1>:8082/artifactory/api/search/aql|200|242|0|9
artifactory-request-out.log:102:2024-12-27T08:31:32.880Z|7c5824a5cf507379|test-generic-remote-3|admin|POST|http://<art_node1>:8082/artifactory/api/search/aql|200|242|0|11
artifactory-request-out.log:103:2024-12-27T08:31:32.882Z|7c5824a5cf507379|test-generic-remote-6|admin|POST|http://<art_node1>:8082/artifactory/api/search/aql|200|242|0|13
artifactory-request-out.log:104:2024-12-27T08:31:32.883Z|7c5824a5cf507379|test-generic-remote-7|admin|POST|http://<art_node1>:8082/artifactory/api/search/aql|200|242|0|14
artifactory-request-out.log:105:2024-12-27T08:31:32.883Z|7c5824a5cf507379|test-generic-remote-5|admin|POST|http://<art_node1>:8082/artifactory/api/search/aql|200|242|0|12
artifactory-request-out.log:106:2024-12-27T08:31:32.945Z|7c5824a5cf507379|test-generic-remote-5|admin|HEAD|http://<art_node1>:8082/artifactory/test-generic-local-5/test5.file|200|0|2|4
artifactory-request-out.log:107:2024-12-27T08:31:32.945Z|7c5824a5cf507379|test-generic-remote-3|admin|HEAD|http://<art_node1>:8082/artifactory/test-generic-local-3/test3.file|200|0|2|3
artifactory-request-out.log:108:2024-12-27T08:31:32.951Z|7c5824a5cf507379|test-generic-remote-2|admin|HEAD|http://<art_node1>:8082/artifactory/test-generic-local-2/test2.file|200|0|2|10
artifactory-request-out.log:109:2024-12-27T08:31:32.976Z|7c5824a5cf507379|test-generic-remote-2|admin|GET|http://<art_node1>:8082/artifactory/test-generic-local-2/test2.file:properties|302|0|0|3
artifactory-request-out.log:110:2024-12-27T08:31:32.979Z|7c5824a5cf507379|test-generic-remote-6|admin|HEAD|http://<art_node1>:8082/artifactory/test-generic-local-6/test6.file|200|0|2|6
artifactory-request-out.log:111:2024-12-27T08:31:32.984Z|7c5824a5cf507379|test-generic-remote-7|admin|HEAD|http://<art_node1>:8082/artifactory/test-generic-local-7/test7.file|200|0|2|5
artifactory-request-out.log:112:2024-12-27T08:31:32.984Z|7c5824a5cf507379|test-generic-remote-2|admin|GET|/artifactory/api/storage/test-generic-local-2/test2.file?propertiesXml|404|0|0|4

     

 

  • We can see from the Artifactory node 2 logs, the test-generic-virtual only sent AQL queries to 5 of the local repositories, resulting in incomplete files and downloads. 
     
Resolution 

The Artifactory AQL API facilitates a transitive search capability, enabling the querying of both cached and non-cached items. However, when conducting an AQL search against a virtual repository, the scope is restricted to the top five remote repositories that it aggregates.  This limitation is in place to ensure optimal performance.

You can adjust this limit by adding a system property to the $JFROG_HOME/artifactory/etc/artifactory/artifactory.system.properties file to set a new limit, for example 10. 
artifactory.aql.transitive.remote.repos.limit=10



Restart the Artifactory node so the configuration takes effect.