ARTIFACTORY: Migration to S3

Nir Shervi
2022-08-10 13:45

Introduction:

Artifactory fully supports S3 object storage for distributed file systems so your Artifactory filestore can reside in the cloud. For more benefits, you can visit JFrog Documentation.
This Knowledge base article is for Artifactory 7.x.

Prerequisites:

1. Prior to the migration, make sure to backup your data, you may use Artifactory’s backup mechanism. More information can be found on our wiki page.

2. Make sure an S3 bucket is available and configured.
 

Procedure:

There are two ways to migrate your filestore over to your S3 provider:

Automatically:

1. Stop Artifactory
2. Create “eventual” and “_pre” directories, under $JFROG_HOME/artifactory/var/data/artifactory/:$ mkdir $JFROG_HOME/artifactory/var/data/artifactory/eventual
$ mkdir $JFROG_HOME/artifactory/var/data/artifactory/_pre

3. CD into the “eventual” directory, and create 2 symbolic links, _add and _pre. They should link to “filestore” (_add) and “pre”(_pre) directories we just created:$ cd $JFROG_HOME/artifactory/var/data/artifactory/eventual
$ ln -s ../filestore/ _add
$ ln -s ../_pre/ _pre

4. Make sure that the system links were created, listed inside “eventual” and pointing to the correct targets (in the previous directory):$ ls -la $JFROG_HOME/artifactory/var/data/artifactory/eventual
total 8
drwxr-xr-x 2 root root 4096 Sep  8 10:01 ./
drwxr-x--- 8 root root 4096 Sep  8 09:58 ../
lrwxrwxrwx 1 root root   13 Sep  8 10:01 _add -> ../filestore
lrwxrwxrwx 1 root root    8 Sep  8 10:01 _pre -> ../_pre

5. Assign the right permissions to the directory created in the first step, assign the user and group that owns your Artifactory installation, by default artifactory:artifactory:$ chown  -R artifactory:artifactory
$JFROG_HOME/artifactory/var/data/artifactory/eventual
$ ​​chown  -R artifactory:artifactory
$JFROG_HOME/artifactory/var/data/artifactory/_pre

6. Configure the binarystore.xml with the S3 bucket. The example below shows an S3 bucket configuration : (for more S3 buckets – click here)<config version="2">
     <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider id="eventual" type="eventual">
                <provider id="retry" type="retry">
                    <provider id="s3-storage-v3" type="s3-storage-v3"/>
                </provider>
            </provider>
        </provider>
    </chain>
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>nirsh-artifactory-s3bucket</bucketName>
       <path>pathPrefix</path>
       <region>us-east-3</region>
       <useInstanceCredentials>true</useInstanceCredentials>
    </provider>
</config>​​​​​​

7. Start  Artifactory.
8. The '../eventual/ add' folder should be empty to indicate that the migration is complete.
9. After the migration is completed, you can replace the template from step six to work as S3 Direct Upload Template – you can read about it and why is it recommended in the JFrog documentation .<config version="2">
   <chain template="s3-storage-v3-direct"/>
   <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <usePresigning>true</usePresigning>
       <signatureExpirySeconds>600</signatureExpirySeconds>
   </provider>
</config>

Note: if you require to create _delete file in the eventual directory, do it and give it permissions.

Manual migration:

If you encounter any difficulties during the automatic migration to S3, you may also try to migrate manually following the steps in the JFrog Documentation.

When you perform manual migration you must stop Artifactory, manually copy the filestore from its current location to the S3 Bucket, configure binarystore.xml using one of our s3 template, and then restart Artifactory.
You can copy the files and store them in the bucket with AWS CLI.

You can check the transfer was successful by downloading an artifact from Artifactory after performing the migration.
 

Troubleshooting:

Edit the $JFROG_HOME/artifactory/etc/artifactory/logback.xml file to increase log verbosity, the example shown below creates an appender with a DEBUG verbosity for the filestore/binarystore mechanism named‘artifactory-filestore.log’
in the logs file:

Note: this process does not require Artifactory to be restarted.<appender name="filestore" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <File>${log.dir}/artifactory-filestore.log</File>
  <rollingPolicy class="org.jfrog.common.logging.logback.rolling.FixedWindowWithDateRollingPolicy">
    <FileNamePattern>${log.dir.archived}/artifactory-filestore.%i.log.gz</FileNamePattern>
    <maxIndex>10</maxIndex>
  </rollingPolicy>
  <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
    <MaxFileSize>25MB</MaxFileSize>
  </triggeringPolicy>
  <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
    <layout class="org.jfrog.common.logging.logback.layout.BackTracePatternLayout">
      <pattern>%date{yyyy-MM-ddTHH:mm:ss.SSS, UTC}Z [jfrt ] [%-5p] [%-16X{uber-trace-id}] [%-30.30(%c{3}:%L)] [%-20.20thread] - %m%n</pattern>
    </layout>
  </encoder>
</appender>
<logger name="org.artifactory.addon.filestore" additivity="false">
  <level value="DEBUG"/>
  <appender-ref ref="filestore"/>
</logger>