Subscription Information
This feature is supported on the Self-Hosted platform, with an Enterprise X or Enterprise+ license.
Overview
JFrog Cold Storage is an archiving solution that enables you to move artifacts or packages not in use from one Artifactory instance to another Artifactory instance that is connected to cost-effective storage. As such, Cold Storage provides customers with the ability to retain artifacts and packages that are not in use, but cannot be deleted for several years due to regulatory obligations.
The Cold Storage solution provides an easy setup and configuration process. In addition, it enables you to define an archiving policy that runs periodically, select artifacts or packages based on specific criteria, and then move the artifacts or packages to a Cold Artifactory instance. The artifacts or packages that are moved to a Cold instance are removed from the Live Artifactory instance. This helps to improve the usability, search capabilities, and performance of the Live instance, by reducing the number of artifacts and packages that require maintenance.
Artifacts and packages that have been archived in the Cold Artifactory instance can be searched for and restored at any point in time using either the UI or dedicated archive search-and-restore APIs. Archived artifacts and packages can be restored to their original location in the Live Artifactory instance or to a new destination. The restore mechanism runs as a background process and notifies you when completed. The metadata associated with archived artifacts or packages is not lost during the archiving period, and you can continue to consume this data after items are restored.
In the following sections, you will find information about how the Cold Storage is set up, as well as the features that are included in this solution.
Note
Currently there is a limitation when using Cold Storage with a Federated Repository. If a retention policy (used to archive artifacts or packages to a Cold Storage server) is configured to clear artifacts or packages from a Federated Repository, the archiving event and any restore event will not be synced with the Federation members. In addition, any deleted artifacts or packages will be recreated due to the Federation. Therefore, it is not recommended to use Cold Storage with Federated Repositories.
In a future Artifactory version, the sync with the Federation members will be enabled.
Cold Storage Architecture
Note
For information about configuring Cold Storage, see Setup Methods.
 |
The Cold Storage archiving solution is comprised of three main components:
Live Artifactory instance
Live Artifactory is a standard Artifactory instance from which artifacts or packages will be archived.
Cold Artifactory instance
Cold Artifactory is a stripped-down, limited-functionality version of an Artifactory instance that is used for archiving purposes. The Cold instance is implemented as a dedicated Artifactory instance, connected to a binary storage provider (such as AWS Glacier).
Note
One Cold Artifactory instance can be connected to only one Live Artifactory cluster. Every cluster of Artifactory that needs to archive requires a single Cold Storage instance with a dedicated license.
Archive (binary storage provider)
When setting up the Cold instance, you will need to connect it to a binary storage provider. You can use one of the following storage options:
A cloud provider, such as AWS Glacier (currently does not support package archiving)
Cold Storage Features
Cold Storage has a number of features that enable you to archive, search, restore and monitor the archiving process. In the following section, you will find a detailed overview of the Cold Storage features as well as information about the logs that are created for some of these functions.
Archive Policy
Cold Storage enables admins to create archiving policies, which are used to decide which artifacts or packages need to be archived. When the archiving policy is executed, artifacts or packages are moved from the Live Artifactory instance to the Cold Artifactory instance, while the corresponding binaries are stored in the storage connected to the Cold Artifactory.
The archiving policies can be configured using the following options:
Scheduled Execution: Scheduling enables you to configure the periodic execution of a policy that can be set to a daily, weekly, or monthly frequency. You can also define when (or after what period) the policy execution will be stopped, which enables you to perform archiving execution during weekends or maintenance windows.
Note
If required, the archiving policy can also be executed manually.
AQL-based Selection Criteria: When archiving artifacts (not for package archiving), you can define an AQL query that is executed to collect artifacts targeted for archival. The selection criteria can include:
Folders and Repositories: Select artifacts or packages located in specific folders within specific repositories.
Properties: Select artifacts based on specific properties, such as application versioning
Age: Select artifacts based on usage and age by defining 'Last Downloaded before' and 'Deployed before'
After artifacts or packages are successfully archived in the Cold instance, the archived items and corresponding metadata are deleted from the original repository in the Live instance. If the archive process fails, items and the corresponding metadata are not removed, and the incident is logged with an appropriate error message.
As the archive process progresses, the system generates detailed logs in the Artifactory service logs in the Live Artifactory instance. If an email server has been configured, the system sends notifications to the admin after the archiving policy execution is completed.
Admins can create, view, edit, and delete archiving policies as the need arises using the UI or APIs. For more information, see Working with Cold Storage.
Note
Only Artifactory admins can create and run archiving policies.
Cold Storage Archive Search
The search functionality enables admins to perform a dedicated archive search from the Live Artifactory for artifacts or packages that can later be restored. For information about performing search operations, see Search and Restore Archived Artifacts or Packages or to search using the REST APIs see Work with Cold Storage Using REST APIs.
Restoring Archived Items
After performing a search for archived artifacts or packages, admins can trigger the restoration of artifacts or packages from the Cold instance to the Live instance. Restoration is a non-destructive process and none of the restored artifacts or packages are deleted from the Cold instance.
When creating a request for restoration, the admin can choose one of the following destinations:
Restore items to their original repository and provide a fallback repository (if the original repository was deprecated)
Restore all the items to a new repository
The system performs the restore operation in the background and updates the Artifactory service logs in the Cold Artifactory instance. After the restore operation is complete, the admin is notified by an email about the completion. This email provides a summary of the operation.
For more information about restoring artifacts, see Restore Archived Artifacts and Packages or Work with Cold Storage Using REST APIs.
Note
The admin must have an email server configured to receive email notifications.
Depending on the number of items being restored, the restore process can take a few minutes to a few hours.
If the restore process encounters an error, the process fails and corresponding error messages are logged.
Monitor Cold Storage
Policy and Archive Logs
Cold Storage adds dedicated logs in the Live Artifactory service logs for the events that are related to the artifacts and packages being archived, including running archiving policies and the archive process itself. The following events are documented in the log:
Create archiving policy
Perform archiving policy execution
Finish archiving policy execution, which includes the following information:
A short summary about the execution, including information about when the process started and ended, and the success percentage
The number of total processed items, successfully archived items, and items that failed to be archived
The status of each archived artifact or package