How to configure Direct (eventual-less) Upload Mechanism on S3 filestore

Ino Choi
2021-08-02 21:58

Relevant Versions: Artifactory 7.9.0 and above.

The default S3 chain templates rely on an Eventual Upload Mechanism, whereby an upload from a client is considered successful when the full binary has been uploaded to Artifactory. As an alternative, you can set up S3 Filestore to bypass the Eventual Upload Mechanism and directly upload files to S3.

Below is how the Direct Upload Mechanism works and its advantages:

How it works

  • Client uploads an artifact to Artifactory.
  • Artifactory receives and simultaneously uploads to S3.
  • Artifactory finishes uploading the binary to S3
    • Artifactory returns 201 success to the client. 
    • A database entry for the artifact is created.

Advantages

  • No need for persistent storage for the eventual directory.
  • Uploads may be faster due to less contention for disk IO, particularly when Artifactory is hosted on AWS.

To enable Direct Upload Mechanism, apply the following custom chain template in the binarystore.xml file (under $JFROG_HOME/artifactory/var/etc/artifactory/).<config version="2">
<chain>
<provider id="cache-fs" type="cache-fs">
<provider id="s3-storage-v3" type="s3-storage-v3"/>
</provider>
</chain>
<provider type="cache-fs" id="cache-fs">
<maxCacheSize>5000000000</maxCacheSize>
</provider>
<provider id="s3-storage-v3" type="s3-storage-v3">
<endpoint>s3.amazonaws.com</endpoint>
<bucketName>bucketName</bucketName>
<path>pathPrefix</path>
<region>s3Region</region>
<identity>yourIdentity</identity>
<credential>yourCredentials</credential>
</provider>
</config>
After making the change, restart Artifactory.

Note! If you are migrating from any Eventual Mechanism to the Direct Upload Mechanism, make sure your eventual directory is empty or you could experience data loss.

For more information, visit the confluence page on Eventual Versus Direct(eventual-less) Upload Mechanism.