Introduction
When configuring Databricks notebooks to interact with JFrog Artifactory for package downloads or repository authentication, users often provide the Artifactory username and access token directly within the notebook code. While this approach works functionally, it exposes sensitive credentials in plaintext inside notebook cells and scripts, which is not recommended from a security perspective.
To avoid direct credential exposure, Databricks provides a built-in secret management mechanism called Databricks Secret Scopes. This functionality allows users to securely store sensitive information such as usernames, passwords, and access tokens, and then reference them dynamically within notebooks without displaying the actual values.
Using this method ensures that credentials remain concealed from notebook outputs while still allowing successful authentication against JFrog Artifactory repositories.
This article will guide you through the process of securely configuring JFrog Artifactory credentials within a Databricks workspace using Databricks CLI and Secret Scopes.
It will cover the following steps:
- Install and configure Databricks CLI
- Create a secure secret scope
- Store Artifactory credentials as encrypted secrets
- Validate the secret retrieval from the Databricks notebook
- Test successful package access from JFrog Artifactory without exposing credentials in notebook code
Resolution
Step 1: Install Databricks CLI on the Local Machine
Please refer to Databricks official documentation here for downloading Databricks CLI.
Step 2: Configure the Databricks Workspace with CLI Authentication
Configure the Databricks workspace by authenticating the CLI using the following command:
databricks configure --token
You will be prompted to provide:
- The Databricks workspace URL
- Databricks Personal Access Token
The Personal Access Token can be generated from:
Databricks Workspace → User Settings/Developer Settings → Access Tokens
Please ensure that the generated token has sufficient permissions to create and manage secret scopes.
Step 3: Create a Secret Scope for Artifactory Credentials
Once the Databricks CLI is configured, create a new secret scope to securely store the Artifactory credentials.
For this example, the secret scope name used is: jfrog-artifactory
You may modify the scope name according to your internal naming standards.
Run the below command:
databricks secrets create-scope jfrog-artifactory --scope-backend-type DATABRICKS
Step 4: Store the Artifactory Username as a Secret
Store the Artifactory username inside the newly created scope.
Interactive method:
databricks secrets put-secret jfrog-artifactory username
OR direct value method:
databricks secrets put-secret jfrog-artifactory username --string-value "<username>"
Step 5: Store the Artifactory Token/Password as a Secret
Store the Artifactory access token or password similarly.
Interactive method:
databricks secrets put-secret jfrog-artifactory token
OR direct value method:
databricks secrets put-secret jfrog-artifactory token --string-value "<token>"
The saved secret values can be updated in the future by rerunning the same command with a new value.
To verify the stored secret keys and their last updated timestamps, run:
databricks secrets list-secrets jfrog-artifactory
Step 6: Validate Secret Retrieval Inside the Databricks Notebook
Once the secrets are successfully created, validate that the notebook can retrieve them.
For example, in our test we have used the following Python code:
Once the values are saved as secrets, they will never be disclosed inside the notebook.
You can now securely use the credentials in the DataBricks notebook to fetch any package. One such example is given below:
