How should I switch to use the Amazon S3 Official SDK?

Ariel Kabov
2019-08-28 07:49

Relevant Versions: Artifactory 6.12.0 and above.

Starting Artifactory 6.12.0, Artifactory can interact with the S3 Cloud storage provider using the official Amazon SDK.
The previously used JetS3t library can still be used, and in fact, existing Artifactory installations have not been switched to use the new library upon upgrading to a supported version.
(Note: A “forced” migration may occur in future versions)

How to tell which S3 SDK I'm using?
The filestore is configured in the $ARTIFACTORY_HOME/etc/binarystore.xml.
If the configuration uses a provider with type="s3-storage-v3", this means the Official Amazon S3 SDK is being used.
If the type is "s3", Artifactory is still using the JetS3t library.

Why would you want to move away from the JetS3t library?

  1. It is deprecated, the last release was in 2015.
  2. The list of issues is pilling up, including security issues.
  3. To support the latest features available in S3 which were not taken into consideration ad the time JetS3t was developed.
  4. The new SDK is official and supported by Amazon.

How to switch to use the Amazon S3 Official SDK?
Switching over is extremely easy, even if you are not an expert at Configuring the Filestore.

In $ARTIFACTORY_HOME/etc/binarystore.xml we should update as below:

If Exists Replace to
chain template="s3" chain template="s3-storage-v3"
chain template="cluster-s3" chain template="cluster-s3-storage-v3"
provider id="s3" type="s3" provider id="s3-storage-v3" type="s3-storage-v3"
type="s3" type="s3-storage-v3"

After updating the file, a restart of Artifactory will kick-in these changes, and it will switch to use the Amazon official SDK.

Note! If you used any of the dedicated JetS3t properties in the “s3” provider, these will not be honored by the new SDK.
However, check if the parameter is exposed in the documentation, and if so set it in the “s3-storage-v3” provider.

Examples

Old Configuration #1 – simple "s3"
<config version="2">
    <chain template="s3"/>
    <provider id="s3" type="s3">

       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>

Migrated Configuration #1 – simple "s3"
<config version="2">
    <chain template="s3-storage-v3"/>
    <provider id="s3-storage-v3" type="s3-storage-v3">

       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>
</config>

Old Configuration #2 – "s3" with properties
<config version="v1">
    <chain template="s3">
        <provider id="cache-fs" type="cache-fs">
            <provider id="eventual" type="eventual">
                <provider id="retry" type="retry">
                    <provider id="s3" type="s3"/>
                </provider>
            </provider>
        </provider>
    </chain>

    <provider id="cache-fs" type="cache-fs">
        <maxCacheSize>100000000000</maxCacheSize>
    </provider>

    <provider id="eventual" type="eventual">
        <numberOfThreads>10</numberOfThreads>  
        <timeout>180000</timeout>
        <dispatcherInterval>5000</dispatcherInterval>
    </provider>

    <provider id="retry" type="retry">
        <maxTrys>10</maxTrys>
        <interval>1000</interval>
    </provider>

    <provider id="s3" type="s3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
       <property name="s3service.disable-dns-buckets" value="true"></property>                              
       <property name="httpclient.max-connections" value="200"></property>

    </provider>

</config>

Migrated Configuration #2 – "s3" with properties
<config version="v1">
    <chain template="s3-storage-v3">
        <provider id="cache-fs" type="cache-fs">
            <provider id="eventual" type="eventual">
                <provider id="retry" type="retry">
                    <provider id="s3-storage-v3" type="s3-storage-v3"/>
                </provider>
            </provider>
        </provider>
    </chain>

    <provider id="cache-fs" type="cache-fs">
        <maxCacheSize>100000000000</maxCacheSize>
    </provider>

    <provider id="eventual" type="eventual">
        <numberOfThreads>10</numberOfThreads>  
        <timeout>180000</timeout>
        <dispatcherInterval>5000</dispatcherInterval>
    </provider>

    <provider id="retry" type="retry">
        <maxTrys>10</maxTrys>
        <interval>1000</interval>
    </provider>

    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
       <enablePathStyleAccess>true</enablePathStyleAccess>                              
       <maxConnections>200</maxConnections>

    </provider>

</config>

Old Configuration #3 – "cluster-s3"
<config version="2">
    <chain template="cluster-s3">
        <provider id="cache-fs-eventual-s3" type="cache-fs">
            <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
                <sub-provider id="eventual-cluster-s3" type="eventual-cluster">
                    <provider id="retry-s3" type="retry">
                        <provider id="s3" type="s3"/>
                    </provider>
                </sub-provider>
                <dynamic-provider id="remote-s3" type="remote"/>
            </provider>
        </provider>
    </chain> 

    <provider id="cache-fs-eventual-s3" type="cache-fs">
        <maxCacheSize>100000000000</maxCacheSize>
    </provider>
  
    <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
        <redundancy>3</redundancy>
        <lenientLimit>2</lenientLimit>
        <property name="zones" value="local,remote"/>
    </provider>
 
    <provider id="eventual-cluster-s3" type="eventual-cluster">
        <maxWorkers>10</maxWorkers>
        <dispatcherInterval>1000</dispatcherInterval>
        <checkPeriod>15000</checkPeriod>
        <addStalePeriod>300000</addStalePeriod>
        <zone>local</zone>
    </provider>

    <provider id="remote-s3" type="remote">
        <checkPeriod>15000</checkPeriod>
        <connectionTimeout>5000</connectionTimeout>
        <socketTimeout>15000</socketTimeout>
        <maxConnections>200</maxConnections>
        <connectionRetry>2</connectionRetry>
       <zone>remote</zone>
    </provider>

    <provider id="s3" type="s3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>

</config>

Migrated Configuration #3 – "cluster-s3"
<config version="2">
    <chain template="cluster-s3-storage-v3">
        <provider id="cache-fs-eventual-s3" type="cache-fs">
            <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
                <sub-provider id="eventual-cluster-s3" type="eventual-cluster">
                    <provider id="retry-s3" type="retry">
                        <provider id="s3-storage-v3" type="s3-storage-v3"/>
                    </provider>
                </sub-provider>
                <dynamic-provider id="remote-s3" type="remote"/>
            </provider>
        </provider>
    </chain> 

    <provider id="cache-fs-eventual-s3" type="cache-fs">
        <maxCacheSize>100000000000</maxCacheSize>
    </provider>
  
    <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
        <redundancy>3</redundancy>
        <lenientLimit>2</lenientLimit>
        <property name="zones" value="local,remote"/>
    </provider>
 
    <provider id="eventual-cluster-s3" type="eventual-cluster">
        <maxWorkers>10</maxWorkers>
        <dispatcherInterval>1000</dispatcherInterval>
        <checkPeriod>15000</checkPeriod>
        <addStalePeriod>300000</addStalePeriod>
        <zone>local</zone>
    </provider>

    <provider id="remote-s3" type="remote">
        <checkPeriod>15000</checkPeriod>
        <connectionTimeout>5000</connectionTimeout>
        <socketTimeout>15000</socketTimeout>
        <maxConnections>200</maxConnections>
        <connectionRetry>2</connectionRetry>
       <zone>remote</zone>
    </provider>

    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>http://s3.amazonaws.com</endpoint>
       <identity>[ENTER IDENTITY HERE]</identity>
       <credential>[ENTER CREDENTIALS HERE]</credential>
       <path>[ENTER PATH HERE]</path>
       <bucketName>[ENTER BUCKET NAME HERE]</bucketName>
    </provider>

</config>