S3 Cluster Binary Provider

This is the setting used for S3 Official SDK Template Amazon S3 Official SDK Template library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.

When using the cluster-s3-storage-v3 template, data is temporarily stored on the file system of each node using the Eventual-Cluster Binary Provider, and is then passed on to your S3 object storage for persistent storage.

Each node has its own local filestore (just like in the file system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providersusing the Sharding-Cluster Binary Provider .

cluster-s3-storage-v3 template configuration

Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-s3-storage-v3 template, your binarystore.xml configuration file should look like the example below.

<config version="2">
    <chain template="cluster-s3-storage-v3"/>
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <usePresigning>true</usePresigning>
       <signatureExpirySeconds>600</signatureExpirySeconds>
   </provider>
</config>

What's in the cluster-s3-storage-v3 template?

While you don't need to configure anything else in your binarystore.xml, this is what the cluster-s3-storage-v3 template looks like under the hood.

<config version="2">
    <chain> <!-- template="cluster-s3-storage-v3" -->
      <provider id="cache-fs-eventual-s3" type="cache-fs">
         <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
            <sub-provider id="eventual-cluster-s3" type="eventual-cluster">
               <provider id="retry-s3" type="retry">
                  <provider id="s3-storage-v3" type="s3-storage-v3"/>
               </provider>
            </sub-provider>
           <dynamic-provider id="remote-s3" type="remote"/>
         </provider>
      </provider>
    </chain>
    <provider id="cache-fs-eventual-s3" type="cache-fs">
        <maxCacheSize>5000000000</maxCacheSize>
    </provider>
    <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
         <redundancy>2</redundancy>
         <lenientLimit>1</lenientLimit>
         <property name="zones" value="local,remote"/>
    </provider>
    <provider id="eventual-cluster-s3" type="eventual-cluster">
         <maxWorkers>5</maxWorkers>
         <dispatcherInterval>1000</dispatcherInterval>
         <checkPeriod>15000</checkPeriod>
         <addStalePeriod>300000</addStalePeriod>
         <zone>local</zone>
    </provider>
    <provider id="remote-s3" type="remote">
         <checkPeriod>15000</checkPeriod>
         <connectionTimeout>5000</connectionTimeout>
         <socketTimeout>15000</socketTimeout>
         <maxConnections>200</maxConnections>
         <connectionRetry>2</connectionRetry>
         <zone>remote</zone>
    </provider>
</config>