How to Connect the Dots: Creating Complex CI/CD with JFrog Pipelines

As software gets more complex, so do software builds. With applications being composed of multiple services — often developed by separate teams — it can be challenging to automate a unified continuous integration process.

JFrog Pipelines is unique among DevOps CI/CD solutions in empowering developers to create highly complex DevOps Pipeline workflows. Pipelines can be defined with multiple paths, trigger points and trigger types. And pipelines can connect to each other as well, enabling teams to collaborate on interdependent projects.

Moreover, as part of the JFrog Platform difference for CI/CD, Pipelines’ natural integration with Artifactory repository manager, Xray, and Distribution provides the control you need to support and manage those complex builds.

Let’s look at some basic examples in JFrog Pipelines to show how complex pipelines can be built from simple mechanisms:

Single Trigger

In the simplest type of CI/CD workflow, a single trigger launches execution of a single path within a single pipeline. For example, this example pipeline builds a Golang service from a single source repository in GitHub. 

In the Pipelines DSL, we declare a GitRepo resource for the Golang service’s source repo (gosvc_gitRepo). By default, the resource will trigger on a commit to any file in any branch, but if you choose to, you can specify more narrow conditions such as: only certain files, only certain branches, or other repository events.

  - name: gosvc_gitRepo
    type: GitRepo
      path: mygithub/go-service
      gitProvider: demo_github
      files:                        # Optional: only specific files
        include:    Dockerfile
        include:    main.go
      branches:                     # Optional: only specific branches
        include:    master
      buildOn:                      # Optional: specify trigger events
         commit:            true    #   Defaults to true
         pullRequestClose:  true    #   Defaults to false
         releaseCreate:     true    #   Defaults to false


Then we can declare that GitRepo as inputResources in the first step of the pipeline.

  - name: demo_gosvc
      - name: bld_svc
        type: DockerBuild
          affinityGroup: DockerImageBuild
          dockerFileLocation: .
          dockerFileName: Dockerfile
          dockerImageTag: ${run_number}
            - name: gosvc_gitRepo
            - name: demoArt


In the default configuration of a GitRepo resource, any new commit to that source repository triggers execution of the pipeline. Or the GitRepo will trigger according to the conditions you may have specified. 

You can see how a single GitRepo change triggers a simple pipeline in this annotated Pipeline History diagram:

If needed, you can instead trigger on receipt of a webhook, or on a cron time interval. To explore these options, see the documentation topic Triggering Pipelines and Steps.

Multiple Triggers

This more complex pipeline example creates an application composed of a front-end section from npm and a back-end section built from a Maven repository. The code for each are in separate source repositories in GitHub. Each component is built separately, but then combined into a single Docker image.

For this type of continuous integration workflow, when either our front-end or back-end component has changed, we’ll need to rebuild only the changed component. Then we’ll need to produce a new Docker image for the application.

In the Pipelines DSL, this is easy to accomplish when you: 

  • Declare a GitRepo resource for the front-end source repo (fe_gitrepo
  • Declare a GitRepo resource for the back-end source repo (be_gitrepo)

Each of those GitRepo resources can now separately trigger our pipeline at different starting points. In the Pipelines DSL, we declare each of those GitRepo resources as inputResources to trigger the steps that perform the builds.

  - name: demo_application
      - name: bld_frontend
        type: NpmBuild
          npmArgs: --no-progress --no-audit
          sourceLocation: .
          repositoryName: kol-npm-dev-remote
            - name: fe_gitRepo
            - name: demoArt
      - name: bld_backend
        type: MvnBuild
          sourceLocation: .
          mvnCommand: clean install -ntp
          configFileLocation: .
          configFileName: mvn-art-config
            - name: be_gitRepo
            - name: demoArt


The resulting multiple trigger points can be seen in this annotated Pipeline History diagram:

Connecting Pipelines

The prior example showed a single pipeline constructed with two trigger paths to build one of two application components that compose a single application. In theory, you could use this same method of multiple trigger paths for an application with three, ten, or a hundred components. But it would be difficult to manage a single pipeline that’s so complex, especially when different teams are responsible for each component.

Instead, it may be easier to have each team create and manage a pipeline for their own component, then connect them into a “pipeline of pipelines” that ultimately produces a final build of the complete application or a signed release bundle for JFrog Distribution.

In JFrog Pipelines, when a new build completes or is promoted, it’s best DevOps practice for it to output a new BuildInfo to Artifactory. This ensures your builds will be fully traceable through your Artifactory repositories. As part of the JFrog Platform, Pipelines’ natural integration with Artifactory and native steps makes this as easy as saying “Yes, please” in the Pipelines DSL.

For example, our prior example pipelines output these BuildInfo resources:

  • The promoted build of the Golang service (gosvc_promoted_build_info)
  • The promoted build of the application (appl_promoted_build_info)

Both of those BuildInfo resources can be used to trigger the next pipeline in the sequence. In the Pipelines DSL, we declare each of those BuildInfo resources as inputResources to trigger the first step, which creates a Distribution release bundle:

  - name: demo_release_mgmt
      - name: bundle
        type: CreateReleaseBundle
          releaseBundleName: demo_rb
          releaseBundleVersion: v1.0.${run_number}
          dryRun: false
          sign: false
          description: "some random test description"
            - name: gosvc_promoted_build_info
            - name: appl_promoted_build_info
            - name: release_bundle
            syntax: markdown
            content: |
              ## Heading
                * Bullet
                * Points


This is the best practice for chaining pipelines together.

This pipeline triggers when either of the prior pipelines successfully completes a new build, as shown in this annotated Pipeline History diagram:

Pipeline of Pipelines

As you can see, your CI/CD workflows can get as complex as your ambition enables, with many individual pipelines that interconnect across teams.

JFrog Pipelines helps you to keep track of them through its innovative Graph view available in the My Pipelines tab of the JFrog Platform. For example, you can see how all of our above examples interconnect for a unified CI process:

In this view, you can see clearly how a set of pipelines work together, and clarify the path to or from any pipeline within the set.

With JFrog Pipelines, you can build your pipelines as complex as you need them to be without losing track of how they interconnect. If needed, you can even interconnect to Jenkins pipelines in your pipeline chain. When your builds fail due to errors, test results, or critical security vulnerabilities, you can readily trace them through their execution path.

Eager to give it a try? Get started with JFrog Platform for free with a JFrog cloud account!