ARTIFACTORY: How should I switch to using the Google native client?

Amit Turgeman
2022-07-17 15:51

Why should I switch to the native client?

The JetS3t library is no longer maintained. Therefore, this template is being deprecated in Artifactory. Use google-storage-v2 as a solution, which uses the Google native client. The transition should be seamless between google-storage to google-storage-v2, as most parameters are the same between the two providers.

How to tell which client I'm using?

Filestore configuration file location:

  • Artifactory 6.x →  $ARTIFACTORY_HOME/etc/binarystore.xml.
  • Artifactory 7.x →  $JFROG_HOME/artifactory/var/etc/artifactory/binarystore.xml.

If the configuration uses one of the below providers, this means the Google native client is being used already.

  •  type="google-storage-v2"
  •  type="cluster-google-storage-v2" 

Why would you want to move away from the JetS3t library?

  1. It is deprecated. The last release was in 2015.
  2. The list of issues is piling up, including security issues.
  3. To support the latest features available in S3, which were not taken into consideration at the time JetS3t was developed.

Switching to v2:

v1

v2

chain template="google-storage"

chain template="google-storage-v2"

provider id="google-storage" type="google-storage"

provider id="google-storage-v2" type="google-storage-v2"

chain template="cluster-google-storage"

chain template="cluster-google-storage-v2"

After updating the file, a restart of Artifactory will kick-in these changes, and it will switch to use the Google native client.

Note! If you used any of the dedicated JetS3t properties in the google-stoarge provider, these will not be honored by the new Google native client.

However, check if the parameter is exposed in the documentation, and if so, set it in the “google-storage-v2” provider section.

For example:
Using dedicated JetS3t properties:<provider id="google-storage" type="google-storage">
<endpoint>commondatastorage.googleapis.com</endpoint>
<bucketName>BUCKET_NAME</bucketName>
<property name="httpclient.max-connections" value="300"/>
<identity>XXXXXX</identity>
<credential>XXXXXX</credential>
</provider>

Moving to google-storage-v2:<provider id="google-storage-v2" type="google-storage-v2">
<endpoint>commondatastorage.googleapis.com</endpoint>
<bucketName>BUCKET_NAME</bucketName>
<maxConnections>300</maxConnections>
</provider>
In the above case, useInstanceCredentials is not defined.

Therefore, the default value will be assigned, false, and the below flow of the authentication resolution order follows

‘useInstanceCredentials’ == false && save creds file under the default path [arti_home_full_path]/etc/gcp.credentials.json.
 

Examples:

Old Configuration #1 – "google-storage" with properties:
 

<config version="v1">
<chain template="google-storage"/>
<provider id="google-storage" type="google-storage">
<endpoint>commondatastorage.googleapis.com</endpoint>
<path>filestore</path>
<bucketName>BUCKET_NAME</bucketName>
<identity>XXXXXX</identity>
<credential>XXXXXXX</credential>
</provider>
</config>

The below example use the first authentication mechanism.‘useInstanceCredentials’ == true && set the "GOOGLE_APPLICATION_CREDENTIALS" env var

For other methods for resolving the authentication resolution use the below link.
Google storage v2 – Authentication Mechanism

Prerequisite:

  • Creating a service account key → download as json.
  • Copy the service-accunt.json to Artifactory instance. 
  • Exporting the path to service-accunt.json as environment variable:
    • Environment variable name → GOOGLE_APPLICATION_CREDENTIALS
    • Export GOOGLE_APPLICATION_CREDENTIALS=’full/path/to/service-accunt.json

Migrated Configuration #1 – "google-storage-v2" with properties:
 

<config version="v2">
<chain template="google-storage-v2"/>
<provider id="google-storage-v2" type="google-storage-v2">
<endpoint>commondatastorage.googleapis.com</endpoint>
<path>filestore</path>
<bucketName>BUCKET_NAME</bucketName>
<bucketExists>true</bucketExists>
<useInstanceCredentials>true</useInstanceCredentials>
</provider>
</config>

Old Configuration #2 – simple "cluster-google-storage”:
 

<config version="2">
<chain template="cluster-google-storage">
<provider id="cache-fs-eventual-google-storage" type="cache-fs">
<provider id="sharding-cluster-eventual-google-storage"
type="sharding-cluster">
<sub-provider id="eventual-cluster-google-storage" type="eventual-cluster">
<provider id="retry-google-storage" type="retry">
<provider id="google-storage" type="google-storage"/>
<endpoint>commondatastorage.googleapis.com</endpoint>
<path>filestore</path>
<bucketName>BUCKET_NAME</bucketName>
<identity>XXXXXX</identity>
<credential>XXXXXXX</credential>
</provider>
</sub-provider>
<dynamic-provider id="remote-google-storage" type="remote"/>
</provider>
</provider>
</chain>
</config>

Migrated Configuration #2 – simple "cluster-google-storage-2”:
 

<config version="2">
<chain template="cluster-google-storage-v2">
<provider id="cache-fs-eventual-google-storage" type="cache-fs">
<provider id="sharding-cluster-eventual-google-storage" type="sharding-cluster">
<sub-provider id="eventual-cluster-google-storage" type="eventual-cluster">
<provider id="retry-google-storage" type="retry">
<provider id="google-storage-v2" type="google-storage-v2"/>
<endpoint>commondatastorage.googleapis.com</endpoint>
<path>filestore</path>
<bucketName>BUCKET_NAME</bucketName>
<bucketExists>true</bucketExists>
<useInstanceCredentials>true</useInstanceCredentials>
</provider>
</sub-provider>
<dynamic-provider id="remote-google-storage" type="remote"/>
</provider>
</provider>
</chain>
</config>