There is a common use case of finding/collecting artifacts that you want to clean up from your repositories. To achieve this, we can use JFrog AQL (Artifactory Query Language) to search for old artifacts that match your cleanup criteria.
The following article will go through two examples of using an AQL to collect a list of old artifacts. It will go through a generic example, and it will go through a Docker example. The difference arises because Docker artifacts use a layered architecture; a Docker image will be composed of several artifacts, including a manifest and individual layers. The manifest is always downloaded, but layers are downloaded conditionally. For example, if the user downloading the Docker image already has a particular layer locally, Docker will not redownload that layer. So Docker cleanup needs to locate the manifest instead of the individual artifacts and then delete the folder of that tag to clean up a specific image.
Important note: This Docker cleanup procedure is a best-effort search and cleanup. With Docker multi-architectural images (those that utilize list.manifest.json), you may find that this method will clean up certain architectures that have not been downloaded per your criteria. This would leave a list.manifest.json that would reference a certain architecture but would not be able to find it.
For a more complete package cleanup, we recommend utilizing Cleanup Policies which is able to take a more complete picture of different packages' metadata.
The following examples will run an AQL (Artifactory Query Language) query to retrieve artifacts matching your configured criteria. It follows a simple three-step pipeline:
- Query: Generates a JSON response containing the filtered artifact data
- Parse: Takes the JSON and pipes it through the jq utility to extract the artifact file paths.
- Export: Saves the paths into a text file, which can be used/fed into your cleanup scripts.
1. For more generic non-Docker repositories
curl -X POST -uuser:password -H "Content-Type: text/plain" -d 'items.find({"repo":{"$match":"<NON-DOCKER-REPO-NAME>"}, "$and" : [ { "modified": {"$before": "180d"}, "created": {"$before": "180d"}}, {"created_by": {"$nmatch": "_system_"}}], "$or" : [ {"stat.downloaded":{"$before": "180d"}, "stat.downloads": { "$eq": null } } ] } ).include("repo", "path", "name", "created", "modified", "stat.downloaded")' "https://<JFROG_URL>/artifactory/api/search/aql" | jq -r '.results[]| (.repo + "/" + .path + "/" + .name)' > NonDockerImagesTargettedForArchiving.txt
This command finds artifacts that:
- Match the repository name '<NON-DOCKER-REPO-NAME>' repository name. (You should replace this with the repository you want to scan through.)
- Have not been modified in the last 180 days and created before 180 days
- The reason for this condition is that if we deploy a new artifact to this repository and run this command right after, then, without this condition, this command will also delete the newly deployed artifact. You can modify the number of days for this condition to match your needs.
- Are not created by the user '_system_'
- This is for metadata files, which are created by Artifactory
- Has not been downloaded in the past 180 days or has not been downloaded at all
- It returns the full path where the artifact is located on.
After running the curl command, it will save the paths to the artifacts found into a file called 'NonDockerImagesTargettedForArchiving.txt'.
2. For Docker repositories
curl -X POST -uuser:password -H "Content-Type: text/plain" -d 'items.find({"repo":{"$match":"<DOCKER-REPO-NAME>"}, "$or": [{"name":{"$eq":"manifest.json"}, "name":{"$eq":"list.manifest.json"} } ], "$and" : [ { "modified": {"$before": "180d"}, "created": {"$before": "180d"} }], "$or" : [ {"stat.downloaded":{"$before": "180d"}, "stat.downloads": { "$eq": null } } ] } ).include("repo", "path", "name", "created", "modified", "stat.downloaded")' "https://<JFROG_URL>/artifactory/api/search/aql" | jq -r '.results[]| (.repo + "/" + .path)' > DockerImagesTargettedForArchiving.txt
This command finds artifacts that:
- Match the repository name '<DOCKER-REPO-NAME>'. (Replace this with your docker repository's name.)
- Have a name of 'manifest.json' or 'list.manifest.json'
- Have not been modified in the last 180 days and created before 180 days
- The reason for this condition is that if we deploy a new artifact to this repository and if we run this command right after, then without this condition, this command will also delete the newly deployed artifact. You can modify the number of days for this condition.
- Has not been downloaded in the past 180 days or has not been downloaded at all
- It returns the folder name where it finds the manifest.
After running the curl command, it will save the path of the images finds the manifests into a text file called 'DockerImagesTargettedForArchiving.txt'.
3. For cleanup
Now that you have the file paths or (in Docker's case) the image folder paths, you can run scripts to clean up those locations.
To do this, you can run a loop against the generated file to iterate through each line and use the Delete Item REST API or the JFrog CLI delete command to iterate through and delete those items. You can split up the file if needed to perform the delete in batches.
I.e.
Delete Item REST API:
curl -XDELETE -H "Authorization: Bearer <TOKEN>" "https://<JFROG_URL>/artifactory/$line"
JFrog CLI delete:
jf rt del $line --quiet
For the JF CLI command, I utilize the '--quiet' flag to skip the confirmation prompt to confirm the deletion.