ARTIFACTORY: S3 bucket content deleted by Artifactory from backend

Introduction 

S3 bucket content deleted by Artifactory from the backend, which leads to artifacts still existing in the UI but cannot be downloaded.

Background and Reason

A customer uses the below binarystore.xm format to connect s3, that is, there are 2 s3 shards, one is read-only and the other is writable, and the redundancy is 1.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<config version="2">
    <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
    </chain>
    <provider id="sharding" type="sharding">
       <redundancy>1</redundancy>
    </provider>
    <provider id="s3-shard1" type="state-aware-s3">
        <bucketName>A</bucketName>
    </provider>
    <provider id="s3-shard2" type="state-aware-s3">
        <bucketName>B</bucketName>
        <writeEnabled>false</writeEnabled>
    </provider>
</config>

In this situation, Artifactory only stores one binary copy and stores it into bucket A (writable) when uploading artifacts. Bucket B keeps empty.

After that, the customer enables bidirectional synchronization(including events for adding and deleting) between bucket A and B at s3 side in the backend, which leads to binary replication from bucket A to bucket B.

Then a large number of binaries are deleted from S3 buckets. The log shows that some binaries are deleted by Artifactory from bucket B, then the other binary copies in bucket A are also deleted due to S3 event replication (the bidirectional synchronization in the backend).

The reason is that Artifactory finds 2 copies for each binary in the two buckets and redundancy = 1, in the next balance cycle Artifactory will delete one of them by default, then the S3 event replication will delete another copy which leads to both copies being deleted.

Resolution

The Artifactory balance between S3 shards is controlled by parameter ‘disableBalanceOnAddStream’ and the default value is false.  To set it to true in the binartstre.xml.(under $JFROG_HOME/artifactory/var/etc/artifactory/) will disable the Artifactory balance and resolve this issue.

For example:
###   
 <provider id="sharding" type="sharding">
        <disableBalanceOnAddStream>true</disableBalanceOnAddStream>
 </provider>
###

 

Note: The default value of redundancy is 1 if there is no explicit set in the binarystore.xml.

It needs to restart Artifactory to make it effective, which will disable the balance check to avoid deleting binary in the next balance if you want to keep artifacts’ binary copies more than the value of redundancy.