Maintaining our repositories and deleting unnecessary binaries is extremely important, as such may reduce costs, increase performance and contribute to keeping the data organized.
The below method, using JFrog CLI and file specs with AQL proves to be one of the fastest approaches to cleanup a large number of artifacts as it is multithreaded, while keeping the stability of the service by not overloading the Artifactory Database, as we are running in batches with a defined limit for each query.
Step one:
Prepare a filespec JSON file:
{
"files": [
{
"aql": {
"items.find": {
"repo": "frontend-generic-dev-local",
"path": {"$match":"*"},
"name": {"$match":"*"},
"type": "file"
}
},
"limit": 50000
}
]
} 1. The limit of 50,000 is the maximum limit we suggest to use to not overload the DB. This means it will search up to 50,000 items to delete in each iteration.
2. You may use any AQL fields of the "item" domain, such as "created", "modified", etc.
Step two:
Prepare a small bash script to run JFrog CLI in a loop. You should calculate the number of iterations required, based on the number of files in the repository and the limit configured above:
#!/bin/bash
for i in {1..1000}
do
echo $i
jfrog rt del --dry-run=true --quiet --threads=8 --spec=aql.json
done *You may increase the number of threads, we do not recommend exceeding 30 threads to not impact other processes using the JFrog platform.Step three:
As the cleanup can take a while, you will most probably want to send it to run in the background. We can utilize the nohup command to do so:
nohup ./script.sh &
Additional notes:
1. Make sure to set "--dry-run=false" for the actual run to start.
2. Before executing the loop on the entire repository, make sure this achieves what you wish and that you are deleting the content you truly intend to delete.
3. Maintaining the cleanup of the repositories might delete artifacts permanently. Please use the scripts with caution.