ARTIFACTORY: How to set up and debug GCS authentication with service account in Kubernetes

Introduction 

There are three methods to configure authentication with GCS in Artifactory.:

Authentication Resolution Order

1. ‘useInstanceCredentials’ == true && set the "GOOGLE_APPLICATION_CREDENTIALS" env var

   
   

2. ‘useInstanceCredentials’ == true && use Kubernetes (or other) service account creds (no creds file)

   
   

3. ‘useInstanceCredentials’ == false && save creds file under the default path 
   $JFROG_HOME/artifactory/var/etc/artifactory/gcp.credentials.json

    

In a Kubernetes environment, if an authentication file is not found in the default location or specified through an environment variable, the system will use the default service account associated with the Artifactory pod to impersonate the appropriate Google service account for accessing GCS. 

Setup 

First, you need to create a google service account if there isn’t one.

gcloud iam service-accounts create ${GSA} --description="" --display-name=${GSA}

 

Next, you need to enable workload identity federation on your kubernete cluster. 
Assume you already have a cluster on GKE, you can check if Workload Identity is enabled:

gcloud container clusters describe ${GKE_CLUSTER_NAME} --location=<cluster location> | grep -A 1 workloadIdentityConfig 

You should see:  

workloadIdentityConfig:
  workloadPool: "PROJECT_ID.svc.id.goog"

Next, we need to create/identify the kubernete service account

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}'

Then, annotate the KSA with the following command:  

kubectl annotate sa ${KSA} iam.gke.io/gcp-service-account=${KSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com

 

We need to establish an IAM policy binding between the `roles/iam.workloadIdentityUser` role and our GSA using the following command:

gcloud iam service-accounts add-iam-policy-binding ${GSA}@${GCP_PROJECT_ID}.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA}]"

Finally, we need to make sure useInstanceCredentials is enabled in filestore config. 


Debug 

If you have encountered an issue with the setting, you can follow the method below to test:

Create a dummy pod to :

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  serviceAccountName: test-ksa
  containers:
  - name: app-container
    image: google/cloud-sdk:slim
    command: ["sh", "-c", "gcloud auth list && gcloud projects list"]

Then we check the logs:

kubectl logs pods/test-pod

We should find something similar in them:

Credentialed Accounts

ACTIVE ACCOUNT
* test-gsa@<your GCP project ID>.iam.gserviceaccount.com
To set the active account, run:
  $ gcloud config set account `ACCOUNT`
PROJECT_ID NAME PROJECT_NUMBER
<your GCP Project ID> <Name of your GCP Project> <Your Project Number>

This means that the KSA has successfully interacted with GCP by impersonating its GSA counterpart. 

You can also try retrieving the service account token from a running pod:

kubectl exec -it <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token -n namespace

Use curl to make request to the metadata server, including the KSA token in the header:

TOKEN=$(kubectl exec -it <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token -n namespace)

curl -H "Authorization: Bearer $TOKEN" -H "Metadata-Flavor: Google" "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token" -n namespace

## The IP address 169.254.169.254 is used by cloud providers like Amazon Web Services (AWS), Google Cloud (GCP), and Microsoft Azure as a reserved endpoint for accessing instance metadata