One of the key features of JFrog Pipelines is the ability to create interconnected pipelines, called a pipeline of pipelines, that can span several different teams, technology platforms, and tools, enabling teams to collaborate on interdependent projects.
Consider the different stages of the SDLC, such as dev, test, deployment to staging, deployment to prod, and so on, with each team creating tens or even hundreds of individual software pipelines. Pipeline of pipelines enables you to easily connect these pipelines to create an end-to-end workflow and exchange information across the pipelines. There is no limit to the number of pipelines that can be interconnected. Pipeline of pipelines can be easily created using global resources, where a pipeline updates a resource, which is used as the input for triggering downstream pipelines.
After a pipeline of pipelines is created, you can view the end-to-end workflow in real-time, interactive Pipelines dashboard, with alerts and notifications, to easily identify and escalate bottlenecks and failures.
Create Pipeline of Pipelines
To create a pipeline of pipelines, you need:
At least two pipelines
A resource that connects these pipelines
Just like how dependent steps are connected using resources, dependent pipelines are also connected using resources.
In the image below, the first pipeline (app_dev_pipeline) ends with the output resource app_buildinfo
. The second pipeline (app_prod_pipeline), which starts with the same resource, uses it as an input resource.
Note
By default, during a run, a dependent pipeline is triggered only when there is a new version of the resource that connects the pipelines.
Examples
Here are two examples showing pipeline of pipelines.
Example 1
This example shows a pipeline of pipelines containing eight pipelines, interconnected using the PropertyBag resource.
In this example:
There are three PropertyBag resources:
There are eight interconnected pipelines:
The pipeline named
TRIGGER
generates an output resource namedrunpopprop
.The resource
runpopprop
is an input resource for these pipelines:TRIGGER_OUT_1_1
TRIGGER_OUT_1_2
TRIGGER_OUT_1_3
TRIGGER_OUT_1_4
The
TRIGGER
pipeline uses write_output utility function to trigger these four pipelines.The resource
runpopprop2
connects the pipelinesTRIGGER_OUT_1_2
andTRIGGER_OUT_1_4
to the pipelineTRIGGER_OUT_3_2
, but does not trigger it.Similarly, the output resource
runpopprop3
from the pipelineTRIGGER_OUT_1_4
does not trigger the pipelineTRIGGER_OUT_1_4
.
Your browser does not support the HTML5 video element
Example 2
This example shows a pipeline of pipelines containing five pipelines, interconnected using several types of resources.
In this example:
These resources are used to connected the pipelines: GitRepo, PropertyBag, Image, BuildInfo, ReleaseBundle, DistributionRule.
There are five interconnected pipelines:
Control Pipeline Runs
When configuring a pipeline of pipelines, you may want to specify when a pipeline may run relative to other pipelines connected by resources.
The dependencyMode
tag, which can be defined in the pipelines.configuration
section in the Pipelines YAML, specifies when a pipeline may run relative to other pipelines connected by resources. It has three settings: waitOnParentComplete
, waitOnParentSuccess
, and waitOnChildComplete
. If any of these settings is true, new runs will not be created for resources updated by other pipelines if a run is already waiting with the same resources and steps. So if a pipeline (named Pipeline_A) runs twice consecutively and the following pipeline (named Pipeline_B) has waitOnParentComplete
set as true
, then Pipeline_B will only run once. When the pipelines do run, they will use the latest resource versions.
These tags can be used when you want to ensure that the input resources of each pipeline step are in a steady state (no runs waiting or in progress) before the step starts executing, to prevent scenarios where the artifacts pointed to by resources change in the middle of step execution. This is particularly useful in situations where a pipeline is creating an artifact that will overwrite itself. For example, a Docker image pushed with the 'latest' tag or a file uploaded each time with the same name. In these cases, some artifacts could change after the step is triggered, resulting in inconsistency in what is downloaded.
The dependencyMode
tag has the following optional settings:
Setting | Description | Default |
---|---|---|
| If | False |
| If | False |
| If | False |
pipelines: - name: myPipeline configuration: chronological: true/false # default false dependencyMode: waitOnParentComplete: true/false # default false waitOnParentSuccess: true/false # default false waitOnChildComplete: true/false # default false steps: ...
Note
Though not recommended, if the parent pipeline has either waitOnParentComplete
or waitOnParentSuccess
and the child pipeline has waitOnChildComplete
, the parent pipeline runs first.
Examples
Example 1 - waitOnParentComplete
and waitOnParentSuccess
In this example, the first pipeline A_parent creates a Docker image. The child pipeline A_waitOnParent waits until the parent pipeline A_parent's run completes successfully, since both waitOnParentComplete
and waitOnParentSuccess
are set astrue
for A_waitOnParent.
resources: - name: dependencyModeImage type: Image configuration: registry: myDocker imageName: jfrog/test imageTag: "0" - name: dependencyModeRepo type: GitRepo configuration: gitProvider: myGithub path: jfrog/dependencyMode branches: include: ^{{gitBranch}}$ pipelines: - name: A_parent steps: - name: createImage type: Bash configuration: inputResources: - name: dependencyModeRepo outputResources: - name: dependencyModeImage integrations: - name: myDocker execution: onExecute: - cd ${res_dependencyModeRepo_resourcePath} - docker build -t ${res_dependencyModeImage_imageName}:${run_number} . - docker push ${res_dependencyModeImage_imageName}:${run_number} - write_output dependencyModeImage imageTag=${run_number} - name: A_waitOnParent configuration: dependencyMode: waitOnParentComplete: true waitOnParentSuccess: true steps: - name: pullImage type: Bash configuration: inputResources: - name: dependencyModeImage execution: onExecute: - echo ${res_dependencyModeImage_imageName}:${res_dependencyModeImage_imageTag}
Example 2 - waitOnChildComplete
In this example, the parent pipeline B_waitOnChild waits until the child pipeline B_Child's run completes, since waitOnChildComplete
is set astrue
for B_waitOnChild.
resources: - name: dependencyModePropertyBag type: PropertyBag configuration: runNumber: 0 pipelines: - name: B_waitOnChild configuration: dependencyMode: waitOnChildComplete: true steps: - name: updateProperties type: Bash configuration: outputResources: - name: dependencyModePropertyBag execution: onExecute: - write_output dependencyModePropertyBag runNumber=${run_number} - name: B_Child steps: - name: echoProperty type: Bash configuration: inputResources: - name: dependencyModePropertyBag execution: onExecute: - echo ${res_dependencyModePropertyBag_runNumber} - sleep 5 - name: echoComplete type: Bash configuration: inputSteps: - name: echoProperty execution: onExecute: - echo "complete"
View Pipeline of Pipelines
Your pipeline of pipelines workflow can be as complex as you want it to be, with several individual pipelines that interconnect across teams. The Pipelines Graph view makes it easy to view and navigate such complex pipelines.
The Graph view:
Shows a combined view of all interconnected pipelines
Provides visualization of pipelines and steps
Shows the pipelines, resources, and interconnecting path that a pipeline is using
Provides the ability to drill down into a single pipeline
Provides real-time updates for a run
Shows project-level multi-pipeline map and list of runs
For more information, see Pipelines Graph view.