S3 Sharding

You can implement sharding with multiple S3 buckets. The s3-sharding template is available with Artifactory to configure sharding with S3. A sub-binary provider, state-aware-s3, template is used with the s3-sharding template to implement sharding across multiple S3 buckets.

Warning

Sharding rules such as redundancy and write/read mechanisms are applicable on the new data added to the filestore post the changes to the filestore configuration. For existing data in the filestore, you must copy the data to other new S3 shards.

S3-sharding Template

The s3-sharding template uses the same parameters as Amazon S3 official template except for UrlPresigning, which is not supported.

<chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
</chain>

State-Aware-S3 Binary Provider

This binary provider is not independent and will always be used with S3 sharding. The provider is aware if its underlying S3 bucket is functioning or not. It can also recover from errors (the parent provider is responsible for recovery) with the addition of the checkPeriod field.

Parameter	Description
type	state-aware-s3
checkPeriod	Default: 15000 ms The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point.
writeEnabled	Default: true Enables/disables the write operations for the binary provider. If set to false, the state-aware-s3 provider will continue to serve read requests, so Artifactory can continue to read binaries from that provider. In addition, the garbage collection can continue to clean the deleted binaries from the provider. (Only applicable under a sharding provider.)
zone	The name of the sharding zone the provider is part of (only applicable under a sharding provider).

S3 Sharding Examples

The following examples show different ways to implement S3 sharding.

S3 Sharding Example 1

In the following sample configuration, the filestore is implemented with two S3 shards, one region, and one redundancy.

<config version="2">

 <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">                
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
 </chain>

 <provider id="sharding" type="sharding">
      <redundancy>1</redundancy>
 </provider>
   
 <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data1212</bucketName>
      <path>yon1220d</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>      
 </provider>
 <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>test-tomers-bucket</bucketName>
      <path>yon1220t</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>     
 </provider>
</config>

S3 Sharding Example 2

In the following sample configuration, the filestore is implemented with two S3 shards, one region, and two redundancy.

<config version="2">
 <chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">               
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
    </chain>   

   <provider id="sharding" type="sharding">
      <redundancy>2</redundancy>
   </provider>
   <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket1</bucketName>
      <path>path1</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>      
   </provider>
   <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket2</bucketName>
      <path>path2</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>      
   </provider>
</config>

S3 Sharding Example 3

In the following sample configuration, the filestore is implemented with five S3 shards, two region, and three redundancy.

<?xml version="1.0" encoding="UTF-8"?>
<config version="5">
 <chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">                
                <sub-provider type="state-aware-s3" id="s3-shard1" />
                <sub-provider type="state-aware-s3" id="s3-shard2" />
                <sub-provider type="state-aware-s3" id="s3-shard3" />
                <sub-provider type="state-aware-s3" id="s3-shard4" />
                <sub-provider type="state-aware-s3" id="s3-shard5" />
            </provider>
        </provider>
    </chain>

   <provider id="sharding" type="sharding">
      <redundancy>3</redundancy>
   </provider>
    <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data120</bucketName>
      <path>yond</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
   </provider>
   <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data125</bucketName>
      <path>yont</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <writeEnabled>false</writeEnabled>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
   </provider>
   <provider id="s3-shard3" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data121</bucketName>
      <path>yonb</path>
      <region>us-west-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
   </provider>
   <provider id="s3-shard4" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data122</bucketName>
      <path>yonb2</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
   </provider>
   <provider id="s3-shard5" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data123</bucketName>
      <path>yonb3</path>
      <region>us-west-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection> 
   </provider>
</config>