The goal of the following article is to provide instructions on how to replicate Artifactory DB from one Artifactory instance to another. In this article, we will focus on migrating Artifactory using the Artifactory Database Dump Method. This method replicates the Artifactory database by dumping it into a file and loading that file to another database. To understand this migration operation, we first need to understand the role of the Artifactory database. Let’s take a closer look at how Artifactory and its database interact with each other. Artifactory utilizes a checksum-based storage, meaning that each unique binary is stored and renamed on the filesystem by its checksum once and only once. If a binary is uploaded a second time, Artifactory will not store another physical copy of the binary, but simply reference the same checksum-based file already present in the filestore to the new Artifact created. In order to support this, Artifactory uses the database to map the required references between the logical representation of a file identified by its checksum to its location in a virtual file system.In addition to supporting checksum-based storage, the Artifactory database contains additional information, including:
Properties – key-value entries, part of the artifact metadata File stats – created/modified dates, size, download count, deployer, etc. File checksum itself – MD5, SHA1, SHA2 Security entities – users, groups, permission targets, etc Archive indices – internal structure mapping
As we have seen each binary is located in the filestore (local disk by default) and is identified by its checksum (SHA1), with all the metadata (such as package metadata, artifact names, sizes, creation dates, repo locations, sha256 values, and signatures) saved in the database. Use Case of the article: Original Instance of Artifactory needs to be moved/copied/migrated to another instance. Can be a requirement for an upgrade. Technical limitations: This method can only be used between the same type of databases. Points to keep in mind: We will be migrating only the DB. The Filestore will need to be migrated as well for a full migration.Important:This is a one-time operation that replicates the database of one Artifactory to another Artifactory server. Any modifications or new artifacts added to the source database instance after the DB dump is done will not appear on the target. To avoid discrepancies of Artifacts between the instances, it will be required to schedule a maintenance window, shutdown Artifactory from the moment you take the DB dump till the end of the process. If you can not afford downtime, consider using this process in conjunction with our other migration options. For example, without any downtime this process can be used for the initial bulk of the migration followed by repository replication to sync the missing delta of Artifacts uploaded after the DB dump. Using a Snapshot: Database migration can be easily done as well using a snapshot of the Database, deploying that snapshot to a new DB of the same type, and connecting the newly created Artifactory to it. Below is an overview of the procedure we will follow.Since Postgres is our recommended DB, we will provide instructions on how to perform this operation with Postgres This migration can be done also with a non-Postgres database. In that case, use the specific dump tool of that DB type. Naming: Source Artifactory - The current original Artifactory. Target Artifactory - New Artifactory that will be created, and migrated to. Steps:
1. Create an instance to host Artifactory of your choice using one of the supported OSs. (If not already created) (Since we are migrating, you have the option to transition to a Kubernetes or a containerized solution as well). 2. Install Artifactory with the exact same version of the original Artifactory on the Target Artifactory, but do not start Artifactory service yet. 3. Using the same Master key: Copy the master.key from the Source Artifactory to the Target. Since you will be using the same DB data, you will need to decrypt it with the same master key as the source Artifactory instance master.key. The master key can be found at : ($JFROG_HOME/artifactory/var/etc/security/master.key - Create the security directory if not present.) 4. Filestore: There are two different options to choose from at this point in the migration process concerning the Filestore migration: A - Creating a new filestore on the target Artifactory and copying the binaries from the source filestore to the new filestore ( rsync or scp can be used, or any other compatible copying method). (If it is a mount or local filestore - Ensuring it is mounted at the same directory location used by the source Artifactory) We recommend doing an initial sync at this stage to transfer the majority of binaries. However we must make sure to make a final quick sync after the DB dump is taken (step 6) to ensure the latest binaries are available on the target Artifactory. B - In the case of a remote filestore (E.g. AWS S3, GCS), we can point the Target Artifactory to the same filestore as the source by changing the remote endpoint name in the binarystore.xml file.Note:Option B should only be done if we intend to decommission the source Artifactory at the end of the migration. We do not support two independent running Artifactory instances pointing to the same filestore except during this migration process, any time after this could lead to unexpected behaviour for both Artifactories. If you wish to have both Artifactories up and working simultaneously please use option A. 5. Make sure the binarystore.xml file points to the correct filestore which has been copied or synced from the original instance. The binarystore.xml configuration file can be found at $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml DB Dump Phase: 6. If a snapshot of the source database can be taken:
-Create a new database from that snapshot -Skip to Step 9
If a snapshot cannot be taken, we will load a dump generated by the Source Artifactory to the Target Artifactory as below:
-Create a new Artifactory database for the target Artifactory of the same type, and set up a database user with appropriate privileges.
For example, in PostgreSQL: CREATE USER artifactory WITH PASSWORD 'password'; CREATE DATABASE artifactory WITH OWNER=artifactory ENCODING='UTF8'; GRANT ALL PRIVILEGES ON DATABASE artifactory TO artifactory;
7. Make a DB Dump from your Source DB. In Postgresql this can be done as follows: pg_dump -h localhost -U username -d database_name -f db_dump.sql For example: pg_dump -h localhost -U artifactory -d artifactory -f db_dump.sql
8. Copy the dump file to the Target Artifactory DB and Load the DB Dump.
Please note that to load the Dump, superuser privileges might be required.
psql -U username -d new_database_name -f db_dump.sql For example: psql -U artifactory -d artifactory -f db_dump.sql
Then your databases should be identical. 9. Configuration file update: Copy your system.yaml from the original instance, while making sure to modify the unique values of your server (Host, IP, etc) Then make sure to edit the DB URL to point to the new DB for this instance. 10. Start Artifactory service, and take a look at the Artifactory service log for a proper boot of Artifactory, and head to the UI to validate the operation. (Give Artifactory extra time to boot). In case of encountering an error: “Your collation seems to have changed” following a DB migration, you can follow the steps in this article to resolve it You're welcome to reach out to JFrog Support for further information or assistance.