For below ~20,000 artifacts, calling the "api/copy" or "api/move" endpoints on a folder should be able to process the request within an acceptable length of time. For larger sets of artifacts, however, these single folder-level APIs do not scale very well. The same is true for the UI buttons that perform the same action.
This is because the direct Copy or Move APIs in Artifactory are single threaded. This was done to ensure that a large copy or move would not bring down the application, but it limits the throughput of the action. An external tool is necessary to do some of the processing.
There exists an alternative multi-threaded approach that should be used when you need to copy or move more than 20,000 artifacts in a timely manner.
The JFrog CLI's Copy / Move Actions Explained
The JFrog CLI has built-in commands which can run these operations using a multithreaded algorithm. The CLI does these steps to perform the action:
-
Search via AQL for the repository and folder path to build a tree of artifacts to copy or move.
-
Run individual copy / move operations, using one thread per artifact found
This results in a lot of individual, file-level actions taking place within a shorter timespan. Processing time is saved because the JFrog CLI is handling the "job", and Artifactory only has to handle the SQL updates.
Note: Increasing the thread count will result in heavier database usage as many more "INSERT" SQL statements will be issued. To improve performance while maintaining stability, monitor the database and increase its resources as necessary.
Example Commands:
Copy the "com" folder with 25 threads
jf rt cp --threads=25 libs-snapshot-local/com libs-release-local/com
Move the "com" folder with 25 threads
jf rt mv --threads=25 libs-snapshot-local/com libs-release-local/com
More Information:
“Move Item” REST API
JFrog CLI “Move Item” command