ARTIFACTORY: AWS S3 connectivity and troubleshooting steps

Pranav Hegde
2022-09-01 11:15

Scenario 1:

Artifactory failed to process the download request due to the S3 filestore permission issue.

Example log: 

2022-03-23T12:59:05.225Z [jfrt ] [ERROR] [7eba4cb41d212e78] [.f.t.s.S3AwsBinaryProvider:185] [http-nio-8081-exec-3] - Failed to download blob  'da0b0dfc939bc14fddd90be244882d3d9b61a567'  from s3
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXXXX4S2TKYQ487Y; S3 Extended Request ID: XXXXX5PPCjwA1Be0frXEcfUaj+dhDnb88woRy4OmZFQ5E3fUS9L5Cmtw31lyyXXXXSx+gS2k=; Proxy: null)

Cause and Solution:

The above error is usually related to wrong credentials or missing permissions. Please check the assigned permissions for connecting to the s3 bucket.
The user should hold the below actions:"s3:PutObject",      
"s3:GetObject",     
"s3:ListAllMyBuckets",      
"s3:AbortMultipartUpload",      
"s3:ListBucket",      
"s3:DeleteObject",      
"s3:HeadBucket",      
"s3:ListMultipartUploadParts"
"s3:CreateBucket"

Scenario 2: 

In some of the scenarios, Artifactory fails to download/upload the artifact since the cache-fs is full.
 

Example screenshot and error log:

User-added image
 

Cause and Solution:

 
Generally, the issue might occur if the storage space allocated for the cache-fs storage is full even though you have enough space in the S3 filestore.
 
The cache-fs works as a binary cache that uses LRU (Least Recently Used) as its cleanup protocol. The cache-fs protocol can improve Artifactory's performance since frequent requests will be served from the cache-fs. The cache-fs binary provider will be the closest filestore layer of Artifactory. This means that if the filestore is mounted, we would like the cache-fs to be local on the artifactory server itself (if the filestore is local, then cache-fs is meaningless).
 
It is good practice to allocate a good amount of size to cache-fs, as it can improve Artifactory's performance since frequent requests will be served from the cache-fs (it uses Least Recently Used as its cleanup protocol). This means that if the filestore is mounted on NFS/ in S3 Cloud Storage, then the cache on local nodes will help to serve frequent download requests received to Artifactory.
 
For example, each time you download an artifact from it will automatically be saved in the cache. It also works as a FIFO (first in first out) order, So when the cache fills up, it will delete the first artifact which was cached.
 
As a rule of thumb, you should monitor for 24 hours the cache-fs and if it gets full in a shorter time than that, it means that the maximum cache size is not enough and can be increased in order to improve the Artifactory performance. For example, if we have 500 GB maximum cache size and after 24 hours the cache was filled only with 200 GB, it will be pretty safe to say that we do not need more than that.
 
When you are using S3 as a filestore with the eventual mechanism, 5GB is the default value for cache-fs provider.<provider id="cache-fs" type="cache-fs">
        <maxCacheSize>50000000000</maxCacheSize>
        <cacheProviderDir>cache</cacheProviderDir>
 </provider>
The maximum storage allocated for the cache in bytes.
  

Scenario 3:

Artifactory failing to come up  when using  “S3 Storage v3”  using encryption key together with testconnection 2022-01-11T10:18:26.955Z [jfrt ] [ERROR] [a1e1ab1fc3221833] [ctoryContextConfigListener:128] [art-init            ] - Application could not be initialized: Failed to test connection with S3AwsBinaryProvider server. Reason found mismatch between remote file ecfd1db9-XXXXXXXX72-c37cd360 and local file: /opt/jfrog/artifactory/var/data/artifactory/testProvider-startup/ecfd1db9-25f2-XXXX-XXXX-XXX
java.lang.reflect.InvocationTargetException: null
Caused by: java.lang.RuntimeException: Failed to test connection with S3AwsBinaryProvider server. Reason found mismatch between remote file ecfd1db9-25f2-41fe-a272-c37cd360 and local file: /opt/jfrog/artifactory/var/data/artifactory/testProvider-startup/ecfd1db9-25f2-XXXXXX-c37cd360a728

Cause and Solution:

This error might be related to an internal known issue, RTMID-18958, “S3 Storage v3 – Using encryption key together with testconnection causes Artifactory load failure” created for a scenario in which using “kmsServerSideEncryptionKeyId” with “testconnection=true” causes Artifactory load failure. In order to resolve this error, you should change the “testconnection” value to “false”, and restart Artifactory”
 

Scenario 4:

Artifactory would throw the below error message when the certificate added for the S3 endpoint is not aligned with the correct DNS name.

Error log:

2022-06-24T10:57:55.232L ^[[1;32m[jfrt ]^[[0;39m ^[[1;31m[ERROR]^[[0;39m [dfebd86567e4587c] [s.S3SignedUrlStorageService:72] [art-init      ] - Failed to check existence of filestore/c1/c1c4e4ae4fe4c264fe253a6b627ccffd5675c808
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <com.XXXXX.artifactory.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com] 

Cause and Solution: 

Based on the error description and explanation that setting up the <useHttp>true</useHttp> was helpful in overcoming the issue. The issue would occur if the cert doesn't contain the failed endpoint. Need to check the DNS associated with the certificate and apply the wildcard related to the certificate.

Scenario 5:

Artifactory will throw the below error message due to the wrong credentials added for the S3 bucket.

 
Error log message: 

2022-04-22 18:37:48,031Z [jfrt ] [WARN ] [b21e8dd09760cb1d] [j.s.S3JetS3tBinaryProvider:478] [art-init      ] - Failed to check if bucket exist, bucket artifactory-replication-storage-s3. Service Error Message.
2022-04-22 18:37:48,036Z [jfrt ] [DEBUG] [b21e8dd09760cb1d] [j.s.S3JetS3tBinaryProvider:388] [art-init      ] - Starting to add item '6c956d8ad21e44e253dc8b8416f02dd39c3a4631' bucket 'artifactory-replication-storage-s3' to s3
2022-04-22 18:37:48,109Z [jfrt ] [ERROR] [b21e8dd09760cb1d] [j.s.S3JetS3tBinaryProvider:399] [art-init      ] - Failed to put blob '6c956d8ad21e44e253dc8b8416f02dd39c3a4631' to s3 caused by 403:The AWS Access Key ID you provided does not exist in our records.
org.jets3t.service.S3ServiceException: Service Error Message.

Cause and Solution: 

This error seems to indicate that is an issue with the credentials provided in the 'binarystore.xml' file. You might want to refer to the AWS documentation below for more information:
https://aws.amazon.com/premiumsupport/knowledge-center/access-key-does-not-exist/
 

Scenario 6:

When the cache is too small or if the cache is full, we would notice the connection timeout issue.
 

Error log message:

Upload … failed due to {}
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool    at org.jfrog.storage.binstore.providers.cache.fs.ConcurrentReadCacheMissHandler.handleCacheMiss(ConcurrentReadCacheMissHandler.java:64) 

 Cause and Solution:

The above error can happen when the cache is too small or if the cache is getting full and the connection times out when Artifactory is trying to query the cache and then buffer the file (since the cache is full). In this case, we need to increase the maxConnections and the connectionTimeout configured in the binarystore.xml.  We need to edit the binarystore.xml with the following (highlighted is my addition) and restart the instance to apply the change.    <useInstanceCredentials>true</useInstanceCredentials>
    <maxConnections>200</maxConnections>
    <connectionTimeout>100000</connectionTimeout>
    <socketTimeout>500000</socketTimeout>
  </provider>

Scenario 7:

Artifactory would throw the below error when it is not able to trust the certificate added for the S3 bucket.

Error log message:

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'storageServiceImpl': Unsatisfied dependency expressed through method 'setBinaryStore' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'binaryServiceImpl': Invocation of init method failed; nested exception is com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target  

Cause and Solution:

The issue happens when the certificates added for the S3 bucket are not trusted by the Artifactory. In order to overcome this, add the required certificate of the S3 bucket to the Artifactory’s $JFORG_HOME/artifactory/var/etc/security/trusted folder and restart the Artifactory for the changes to take effect.
 

Scenario 8:

How to use Amazon s3 bucket with kms server side encryption with artifactory.

Solution:

1. Configure Server-side encryption to the Artifactory S3 bucket. Below link for reference:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html.<config version="2">
    <chain template="cluster-s3-storage-v3"/>
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <usePresigning>true</usePresigning>
       <signatureExpirySeconds>600</signatureExpirySeconds>
       <kmsClientSideEncryptionKeyId>*****</kmsClientSideEncryptionKeyId>
   </provider>
</config>

2. Use KMS details in the binary store template.<config version="2">
    <chain template="cluster-s3-storage-v3"/>
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <usePresigning>true</usePresigning>
       <signatureExpirySeconds>600</signatureExpirySeconds>
       <kmsClientSideEncryptionKeyId>*****</kmsClientSideEncryptionKeyId>
   </provider>
</config>
 

Scenario 9:

Artifactory would throw the below error message when kmsClientSideEncryptionKeyId is configured with the S3 bucket.

Error log message:

2022-03-23T05:12:21.584Z [jfrt ] [ERROR] [45ad7832de304c87] [o.a.w.s.RepoFilter:315 ] [http-nio-8081-exec-3] - Sending HTTP error code 500: null (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: af406bd0-0736-46ba-9410-a9226e9311d9; Proxy: null)
61
com.amazonaws.services.kms.model.AWSKMSException: null (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: af406bd0-0736-46ba-9410-a9226e9311d9; Proxy: null)
60
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
59
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
58
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
57
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
56
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClie

Cause and Solution:

In some of the cases even though the kmsClientSideEncryptionKeyId is correctly configured, if the following URL https://sts.amazonaws.com is not whitelisted on the host/cluster where Artifactory is hosted, you would observe the above error in the Artifactory logs.

Below is the example  I AM permission:{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "XXXXXXXXX"
        },
     {
          "Sid": "AllowUseOfKeyInAccountxxxxx",
          "Effect": "Allow",
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
         "Resource": "XXXXXXXXXX"
       }
    ]
}