ARTIFACTORY: How to Configure Artifactory with S3 Sharding filesystem with two S3 buckets

In Artifactory, we can implement sharding with multiple S3 buckets. 
Through this article, we will illustrate the steps to configure the sharding with two different S3 bucket endpoints.

The s3-sharding template is available with Artifactory to configure sharding logic with S3. A sub-binary provider, state-aware-s3 template is used with the s3-sharding template to implement sharding across multiple S3 buckets. Artifactory filestore configurations are managed in binarystore.xml file which is located in the “$JFROG_HOME/artifactory/var/etc/artifactory” folder. Below are steps to configure two S3 shards with redundancy set to 2.


Step 1

Edit binarystore.xml which is present in $JFROG_HOME/artifactory/var/etc/artifactory” folder, using the following commands.
        -  $ cd $JFROG_HOME/artifactory/var/etc/artifactory
        -  $ vi binarystore.xml (Use the following reference to create the binarystore configuration added with the respective bucket details)
 

<config version="2">
 <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
 </chain>   

   <provider id="sharding" type="sharding">
      <redundancy>2</redundancy>
   </provider>
   <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket1</bucketName>
      <path>path1</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <max.retry.number>2</max.retry.number>
   </provider>
   <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket2</bucketName>
      <path>path2</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <max.retry.number>2</max.retry.number>
   </provider>
</config>

Step 2

Restart Artifactory for the new configuration addition to take effect.
After the restart, Artifactory would be able to add data to both the S3 bucket, identified as shards

Note

Sharding logic such as redundancy and write/read mechanisms are applicable only for the new data deployed to the Artifactory instance. If you already have some data available in the S3 bucket before adding a new bucket as a shard, it is required to manually copy the data to new S3 shards.