Automating your Feature Branch Repository Management with JFrog CLI
By
6 min read
Feature branch workflows are used to isolate work done on a specific feature in a dedicated branch. This allows all development to be kept away from the team’s common codebase until completion. Essentially keeping the master safe from any risk until it is ready to be merged.
The Challenge
Alongside the code itself, developers also need to manage their feature branch artifacts and dependencies. But, do we actually need to manually create a repository for every new feature branch that we work on? How can we maintain that in an easier way, and perhaps automate the whole process of creation and deletion?
In this blog post, we’ll see how to use dedicated artifact repositories in JFrog Artifactory for your feature branch workflows, and easily automate everything with JFrog CLI.
JFrog CLI is a compact and smart client that provides a simple interface that automates access to JFrog products simplifying your automation scripts and making them more readable and easier to maintain. Specifically we’ll see how to manage Artifactory repository automation using JFrog CLI.
Note: this can also be done using REST API and Artifactory User Plugins
The Solution
JFrog CLI offers a set of commands for managing your Artifactory repositories. Including creating, updating and deleting repositories.
The following example demonstrates creating 3 Artifactory repositories (local, remote and virtual), as part of your CI process, using GitHub Actions. This can also be done using any orchestration tool such as JFrog Pipelines, Jenkins, BitBucket, etc…
Learn more on GitHub Actions integration with Artifactory >
In this example, we’ll create a CI process that’s triggered for non-master pull requests. It will include additional steps for automatically creating an Artifactory repository, before building and deploying the software to Artifactory.
The CI process consists of 2 extra steps before the “Build & Deploy” step:
- Feature Branch Repository Creation – creates a new Artifactory repository for the feature branch.
- Feature Branch Repository Update – in case an Artifactory repository has already been created, the “Feature Branch Repository Creation” step will fail, and this step will update the current repository to point to the existing one.
Note: The naming convention for the repositories is important and will be used further to a smart deletion mechanism.
Code outline:
For any branch = X,
|
Code sample:
- if: always()
name: Feature Branch Repository Update
env:
SERVER_ID: tal-eplus-saas
run: |
echo "::set-env name=repository::$(echo ${GITHUB_REF#refs/heads/} | sed 's/\//_/g')"
jfrog rt mvnc --server-id-resolve=$SERVER_ID --server-id-deploy=$SERVER_ID --repo-resolve-releases=auto-cli-virtual-$repository --repo-resolve-snapshots=auto-cli-virtual-$repository --repo-deploy-releases=auto-cli-virtual-$repository --repo-deploy-snapshots=auto-cli-virtual-$repository
- if: always()
name: Build & Deploy
run: jfrog rt mvn clean install -f maths/pom.xml
Advantages
The advantages to using a dedicated feature branch Artifactory repository are:
- Clear isolation of the binaries (artifacts and 3rd party dependencies) you’re using in your development branch.
- Deploy your application without “outside noise” with your dedicated isolated feature branch repository.
- Easy “Cleanup” per feature – once the feature development is done and the branch is merged to ‘master’, all the irrelevant dependencies used for the development / different versions deployed to the feature branch repository can be deleted and tracked easily under one location.
- Security scanning – using JFrog Xray watches on dedicated feature repositories / builds.
Cleanup
To ensure that this solution is scalable for the many features we develop, we need to define a cleanup mechanism to remove any unused old repositories.
The following last step in our example will delete old repositories using the “NUMBER_OF_DAYS_TO_KEEP” parameter. It will delete all 3 repository types (local, remote and virtual), as well as empty repositories. For example, setting “NUMBER_OF_DAYS_TO_KEEP = 90”, will delete all repositories created before 90 days.
We’ll use the following capabilities of Artifactory REST API and the JFrog CLI:
- Retrieving all repositories with ‘auto-cli-local’ prefix – GET /api/repositories.
- Searching with jfrog rt search – to find the last modified file in a repository.
Deleting with jfrog rt rdel – for the deletion of the repository.
Code sample:
- if: always()
name: Feature Branch Repository Deletion
env:
NUMBER_OF_DAYS_TO_KEEP: 90
run: |
# Extract all the LOCAL repositories created automatically by the CI process
jfrog rt curl -XGET /api/repositories | jq '[.[] | .key | select(test("auto-cli-local"))]' > deletion/auto_created_repositories.json && cat deletion/auto_created_repositories.json
# Calculate for which month are we going back to verify who should be deleted
jq -n 'now - 3600 * 24 * '$NUMBER_OF_DAYS_TO_KEEP' | gmtime | todate' > deletion/months_indicator && cat deletion/months_indicator
# Iterate over all the repositories, delete those by the latest file that was modified
jq -c '.[]' deletion/auto_created_repositories.json | while read i; do
echo Iterating repository = $i
sh -c "jfrog rt s --spec deletion/repositories-spec.json --spec-vars='key1="$i"' > deletion/search_results && cat deletion/search_results"
# If the repository is empty / latest modified file is older > NUMBER_OF_DAYS_TO_KEEP days => DELETE the repository
if [[ $(cat deletion/search_results) == "[]" ]]; then
echo "Deleting repository: $i, repository is empty"
sh -c "jfrog rt rdel $i --quiet && jfrog rt rdel ${i//local/virtual} --quiet && jfrog rt rdel ${i//local/jcenter} --quiet"
elif [[ $(cat deletion/search_results | jq --arg month_indicator $(cat deletion/months_indicator) '.[] | .modified | . <= $month_indicator') = "true" ]]; then
echo "Deleting repository: $i, too old to keep in Artifactory"
sh -c "jfrog rt rdel $i --quiet && jfrog rt rdel ${i//local/virtual} --quiet && jfrog rt rdel ${i//local/jcenter} --quiet"
else
echo "Skipping Repository deletion - repository is still relevant"
fi
done
Complete GitHub Actions pipeline:
The full code for this pipeline is available at:
