Blog Home

Automating your Feature Branch Repository Management with JFrog CLI

Tal Yitzhak

By Tal Yitzhak

6 min read

Feature Branch Repository Management

Feature branch workflows are used to isolate work done on a specific feature in a dedicated branch. This allows all development to be kept away from the team’s common codebase until completion. Essentially keeping the master safe from any risk until it is ready to be merged.

Feature Branch Workflows

The Challenge

Alongside the code itself, developers also need to manage their feature branch artifacts and dependencies. But, do we actually need to manually create a repository for every new feature branch that we work on? How can we maintain that in an easier way, and perhaps automate the whole process of creation and deletion?

In this blog post, we’ll see how to use dedicated artifact repositories in JFrog Artifactory for your feature branch workflows, and easily automate everything with JFrog CLI.

JFrog CLI is a compact and smart client that provides a simple interface that automates access to JFrog products simplifying your automation scripts and making them more readable and easier to maintain. Specifically we’ll see how to manage Artifactory repository automation using JFrog CLI. 

Note: this can also be done using REST API and Artifactory User Plugins

The Solution

JFrog CLI offers a set of commands for managing your Artifactory repositories. Including creating, updating and deleting repositories.

The following example demonstrates creating 3 Artifactory repositories (local, remote and virtual), as part of your CI process, using GitHub Actions. This can also be done using any orchestration tool such as JFrog Pipelines, Jenkins, BitBucket, etc…

Learn more on GitHub Actions integration with Artifactory >

In this example, we’ll create a CI process that’s triggered for non-master pull requests. It will include additional steps for automatically creating an Artifactory repository, before building and deploying the software to Artifactory.

Feature Branch Example

The CI process consists of 2 extra steps before the “Build & Deploy” step:

  1. Feature Branch Repository Creation – creates a new Artifactory repository for the feature branch.
  2. Feature Branch Repository Update – in case an Artifactory repository has already been created, the “Feature Branch Repository Creation” step will fail, and this step will update the current repository to point to the existing one. 

Note: The naming convention for the repositories is important and will be used further to a smart deletion mechanism.

Feature Branch Create and Update

Code outline:

For any branch = X,

  1. If X = ‘master’, do nothing.
  2. If X != ‘master’, create 3 repositories (if they don’t exist):
    • 1 local repository: ‘auto-cli-local-X’ for the purpose of saving the resulting artifacts.
    • 1 remote repository: ‘auto-cli-jcenter-X’, pointing to https://jcenter.bintray.com.
    • 1 virtual repository: ‘auto-cli-virtual-X’ pointing to both ‘auto-cli-local-X’ and to ‘auto-cli-jcenter-X’, the CI server will use the URL for this repository.
  3. Update the build’s current repository: ‘auto-cli-virtual-X’ for fetching 3rd party dependencies and pushing the resulting feature branch artifacts.

Code sample:


    - if: always()   
      name: Feature Branch Repository Update
      env:
        SERVER_ID: tal-eplus-saas
      run: |
        echo "::set-env name=repository::$(echo ${GITHUB_REF#refs/heads/} | sed 's/\//_/g')"
        jfrog rt mvnc --server-id-resolve=$SERVER_ID --server-id-deploy=$SERVER_ID --repo-resolve-releases=auto-cli-virtual-$repository --repo-resolve-snapshots=auto-cli-virtual-$repository --repo-deploy-releases=auto-cli-virtual-$repository --repo-deploy-snapshots=auto-cli-virtual-$repository
     
    - if: always()
      name: Build & Deploy
      run: jfrog rt mvn clean install -f maths/pom.xml

Advantages

The advantages to using a dedicated feature branch Artifactory repository are:

  • Clear isolation of the binaries (artifacts and 3rd party dependencies) you’re using in your development branch.
  • Deploy your application without “outside noise” with your dedicated isolated feature branch repository.
  • Easy “Cleanup” per feature – once the feature development is done and the branch is merged to ‘master’, all the irrelevant dependencies used for the development / different versions deployed to the feature branch repository can be deleted and tracked easily under one location.
  • Security scanning – using JFrog Xray watches on dedicated feature repositories / builds.

Cleanup

To ensure that this solution is scalable for the many features we develop, we need to define a cleanup mechanism to remove any unused old repositories. 

The following last step in our example will delete old repositories using the “NUMBER_OF_DAYS_TO_KEEP” parameter. It will delete all 3 repository types (local, remote and virtual), as well as empty repositories. For example, setting “NUMBER_OF_DAYS_TO_KEEP = 90”, will delete all repositories created before 90 days.

We’ll use the following capabilities of Artifactory REST API and the JFrog CLI: 

  • Retrieving all repositories with ‘auto-cli-local’ prefix – GET /api/repositories.
  • Searching with jfrog rt search – to find the last modified file in a repository.

Deleting with jfrog rt rdel – for the deletion of the repository.

Code sample:


    - if: always()
      name: Feature Branch Repository Deletion
      env:
        NUMBER_OF_DAYS_TO_KEEP: 90
      run: |
        # Extract all the LOCAL repositories created automatically by the CI process
        jfrog rt curl -XGET /api/repositories | jq '[.[] | .key | select(test("auto-cli-local"))]' > deletion/auto_created_repositories.json && cat deletion/auto_created_repositories.json

        # Calculate for which month are we going back to verify who should be deleted
        jq -n 'now - 3600 * 24 * '$NUMBER_OF_DAYS_TO_KEEP' | gmtime | todate' > deletion/months_indicator && cat deletion/months_indicator

        # Iterate over all the repositories, delete those by the latest file that was modified
        jq -c '.[]' deletion/auto_created_repositories.json | while read i; do
          echo Iterating repository = $i
          sh -c "jfrog rt s --spec deletion/repositories-spec.json --spec-vars='key1="$i"' > deletion/search_results && cat deletion/search_results"
          
          # If the repository is empty / latest modified file is older > NUMBER_OF_DAYS_TO_KEEP days => DELETE the repository
          if [[ $(cat deletion/search_results) == "[]" ]]; then
             echo "Deleting repository: $i, repository is empty"
             sh -c "jfrog rt rdel $i --quiet && jfrog rt rdel ${i//local/virtual} --quiet && jfrog rt rdel ${i//local/jcenter} --quiet"
          elif [[ $(cat deletion/search_results | jq --arg month_indicator $(cat deletion/months_indicator) '.[] | .modified | . <= $month_indicator') = "true" ]]; then             
             echo "Deleting repository: $i, too old to keep in Artifactory"
             sh -c "jfrog rt rdel $i --quiet && jfrog rt rdel ${i//local/virtual} --quiet && jfrog rt rdel ${i//local/jcenter} --quiet"
          else
             echo "Skipping Repository deletion - repository is still relevant"
          fi
        done

Complete GitHub Actions pipeline:

Complete GitHub Actions pipeline

The full code for this pipeline is available at:

https://github.com/jfrog/project-examples/tree/master/github-action-examples/repo-management-github-actions-example

Popular Tags