Overview
Resources are one of the key building blocks of all pipelines. They are information entities that are used for storing and exchanging information across steps and pipelines.
Resources are versioned and each version is immutable. They are also global and depending on the scope defined for the pipeline source, they can be available across pipelines, which enable you to connect multiple pipelines together to create a pipeline of pipelines.
Resources are pointers and they can be used to reference:
A repository in your source code control system, such as GitHub
A file on a remote file server
A Docker image
A release bundle for JFrog Distribution
A cluster for container orchestration
Using resources in your pipeline involves two main steps:
In the pipeline's YAML, in the
resources
section, define all the resources required for running the pipeline.After a resource is defined, it is available for use in the pipeline, based on the scope defined for the pipeline source.
In the
steps
section, as per your workflow, add these resources as input and/or output.
Resource Types
Resource Scope and Visibility
Pipeline resources have the same scope as the pipeline source where they are defined.
When defining resources, the recommended approach is to define them in a pipeline source that is shared across environments in the same Project. This ensures that the resources are available across environments in a project. For more information, see Creating a Project - Pipelines Resources.
Note
Currently, pipeline resources cannot be shared across Projects.
Resource Types
Resources play different roles based on where and how they are used:
Triggering Resource: Whenever there is a change in this type of resource, it triggers the dependent step. Examples: CronTrigger, GitRepo, IncomingWebhook.
Generated Resource : These are resources that are generated by a step and they can trigger successive downstream steps. Examples: Aql, BuildInfo, FileSpec, Image.
Webhook Resource: IncomingWebhook and OutgoingWebhook are resources that can be used to integrate a pipeline with third-party services.
Pipelines supports several types of resources, with each resource capable of serving a specific activity:
Type | Description |
---|---|
An Aql resource specifies an Artifactory query using Artifactory Query Language. | |
A BuildInfo resource is the metadata associated with a build in Artifactory. | |
A CronTrigger is used as an input resource as a step to trigger execution of the step at a scheduled time. | |
A DistributionRule resource is the set of Destination rules that can be applied to distributing a release bundle using JFrog Distribution. | |
A FileSpec resource specifies a File Spec, which provides the details of files to upload or download from Artifactory. | |
A GitRepo is used to connect JFrog Pipelines to a source control repository. It creates a webhook to the repo so that future commits will automatically create a new version with the webhook payload. | |
The HelmChart resource maps to a specific chart in an Artifactory Helm Repository. | |
An Image resource is used to add a reference to a Docker image to your pipeline. | |
An IncomingWebhook resource can trigger one or more jobs in your pipeline whenever the associated URL is called using the HTTP POST method. | |
An OutgoingWebhook resource uses HTTP to send information from a step to an external API endpoint through an Outgoing Webhook Integration. | |
A PropertyBag resource is used to pass information from one pipeline to another and provide environment variables to a step in the format of a resource. | |
A ReleaseBundle resource specifies a set of artifacts in Artifactory that are distributed to Artifactory Edge nodes as a JFrog Distribution Release Bundle. | |
A RemoteFile resource enables using a file on a remote file server. | |
A VmCluster is used to represent a set of virtual machines. It is mainly used to deploy services/apps to the specified clusters and in some cases, it can be used to run certain maintenance activities on the clusters as a whole. |
Resource Versions
One of the key features of Pipelines resource is versioning. Resource versions are useful for controlling the flow of pipelines and tracking the changes that a resource undergoes over time. You can trigger runs using specific versions or skip steps in the same run when input resources are not updated.
Every resource starts with an initial version, which is updated every time the resource changes. For example, the version of an Image resource used in a pipeline is updated whenever a new tag is pushed to a Docker image. Pipelines tracks these changes by updating the resource version, which is based on the metadata received for that particular resource.
A new resource version is created when:
The resource definition is updated in the pipelines YAML file.
An output resource is updated during a run.
There is an external event, such as pushing a commit to a git repository.
Note
No new version is created for an unchanged resource. However, if you need the resource to always change (for example, to trigger another pipeline), you can add $run_id environment variable as the value of an additional property.
Each version of a resource is immutable and returns the same result every time a specific version is used for a run. By default, steps always run using the latest version of an input resource. However, because a resource is versioned and holds the entire history of all the available versions, a run can be customized to use a specific version of a resource. For more information, see Triggering a Run with Custom Parameters.
Default Behavior of Resource Versions
Resource versions have the following default behavior:
Latest version: When a run is triggered, the latest version of the input resource is used in that run.
Dependent step in the same pipeline: During a run, even when an input resource is not updated (no new version), the dependent step is triggered. While this is the default behavior, it can be changed by setting the
newVersionOnly
tag astrue
.Note
This is applicable only for generated resources, which are resources that connect two steps of a pipeline.
Dependent step in another pipeline: During a run, the dependent pipeline is triggered only when there is a new version of the resource that connects the pipelines.
Example - Resource Versions
This is a simple, single-step pipeline, with GitRepo as the input resource for the step.
In this example:
First version: When the pipeline syncs and loads for the first time, the latest commit on the Git repository is seeded as the very first version for the Gitrepo resource.
When the first run is triggered, this resource version is used for the run.
Second version: When a new commit is pushed to the Git repository, the Gitrepo resource is updated with the new version, which becomes the latest version of the resource.
The Git commit automatically triggers the run and the second version of the resource, which is now the latest, is used for the run.
Creating Resources
All resources are defined in a pipeline YAML under the resources
tag, as shown below. After a resource is defined and committed to a source control, it is consumed within a pipeline, based on the scope defined for the pipeline source.
Sharing resources across environments
When defining resources, the recommended approach is to define them in a pipeline source that is shared across environments in the same Project. This ensures that the resources are available across environments in a project. For more information, see Creating a Project - Pipelines Resources.
While each resource has its own specific configuration, they all require a name
and a type
.
YAML Schema
resources: - name: <string> type: <resource type name> configuration: <as required by type>
Tag | Description of usage | Required/Optional |
---|---|---|
| An alphanumeric string (underscores are permitted) that makes it easy to infer what the resource represents. This name is used to refer to the resource in steps, and must be unique across all repositories in your JFrog Pipelines Project. Example: Note
| Required |
| Name of the resource type that this resource is an instance of. NoteAfter the pipeline performs a sync, its type cannot be modified. | Required |
| Specifies configuration settings, which vary for each Commonly included in this block is a setting that assigns an integrationthrough which the resource will be authenticated and accessed. The integration must be compatible with the | Required |
Examples - Resource Definition
These examples show the YAML definition for GitRepo and Image resources:
Example 1 - GitRepo Resource
resources: - name: gitrepo_trigger type: GitRepo configuration: gitProvider: my_github path: myuser/repo-name branches: include: master
Example 2 - Image Resource
resources: - name: Image_1 type: Image configuration: registry: PSSG_DockerRegistry imageName: docker/jfreq_win imageTag: latest autoPull: true
Modifying Resources
In Pipelines, resources and their versions are tightly coupled. Therefore, when a resource is deleted, its historical data is permanently affected. This can mess up your DevOps Assembly Lines as it is a connected inter-dependent workflow.
The following rules apply when editing resources:
If you modify a resource's name, it is treated as new resource.
A resource's
type
cannot be modified.
If your pipeline is failing because of a modified resource tag,the only option to recover your pipeline is to delete the resource definition. Deletion of a resource is a two-step process:
Using Resources
In a pipeline, steps can use resources as:
Inputs: When a resource is an input for a step, it is called an input resource. The input resource of one step can be the output resource of other steps.
Outputs: When a resource is an output of a step, it is called an output resource. The output resource of one step can be the input resource of other steps.
Input and output resources can originate from the same pipeline source as the pipeline or another pipeline source in the same project and environment.
Input Resources
Input resources enable you to create dependencies between steps and pipelines. Steps that have input resources from other pipelines trigger a run of the pipeline when the resource is updated. By default, input resources that were an output resource of another step in the same run will run whether or not the resource is updated. A resource can also be referred to by its name as an argument in the shell commands that the step executes.
Input Resource Definiton
A resource can be specified as an input for a step by adding it to the inputResources
section of a step.
YAML Schema
steps: - name: <step_name> type: <step_type> configuration: inputResources: - name: <resource name> trigger: <true/false> # default true newVersionOnly: <true/false> # default false branch: <string> # see description of defaults below
Tag | Description of usage | Required/Optional |
---|---|---|
| Name of the declared resource that is to be a used as an input for the step. | Required |
|
| Optional |
| Setting If there are multiple | Optional |
| A | Optional |
Example - Input Resource Definition
steps: - name: step_1 type: Bash configuration: inputResources: - name: my_app_repo trigger: false # optional; default true newVersionOnly: true # optional; default false branch: master # optional
Using Input Resources
This section provides information about the various ways in which input resources can be used to manipulate pipeline runs.
Skip Automatic Trigger for All Commits
By default, changes to an input resource triggers the execution of the dependent steps. For example, when a step specifies a GitRepoGitRepo resource, any new code committed to that Git repository automatically causes that step to execute. However, this behavior can be changed by declaring trigger
as false
(see below). Now, even when the resource is updated, the dependent step is not triggered. This is especially useful for a production pipeline, when you do not want to deploy every new build.
Note
For a step to not be triggered automatically, trigger:
false
must be set for all all input resources in the step.
Note
Even if trigger
is set as false
, the step still receives webhook updates. This ensures that when it is manually triggered, it uses the latest commit.
Example - Automatic Trigger Off
pipelines: - name: java_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: my_app_repo trigger: false - name: cron_trigger trigger: false execution: onExecute: - pushd $res_my_app_repo_resourcePath - ./execute.sh - popd
When trigger
is set as false
, the line linking the input resource and the step appears as a dashed line.
|
| ||
---|---|---|---|
Trigger Automatically on New Version Only
Whenever a resource undergoes a change, its version is updated and the dependent step is triggered. This is the default behaviour for all input resources. To skip steps in a run when input resources are not updated, add the newVersionOnly
tag and set it as true
. During a run, the step is triggered only when the resource is updated. If the resource is not updated, the step is skipped and all the downstream steps are skipped as well.
Example 1 - newVersionOnly
pipelines: - name: java_pipeline steps: - name: step_1 type: Bash configuration: inputResources: - name: my_app_repo - newVersionOnly: true execution: onExecute: - pushd $res_my_app_repo_resourcePath - ./execute.sh - popd
Trigger Manually using Specific Versions
A run can be customized by selecting a specific version for an input resource. For more information, see Triggering a Run with Custom Parameters.
Pinning Resource Versions
By default, Pipelines uses the most recent or latest version of an input resource when running a job. However, there could be cases where you want to use a specific version of an input resource for a run. This is called pinning and input versions can be pinned using the YAML configuration. When a resource version is unpinned, it switches to using the latest version for all subsequent runs.
Resource version Ids have a global sequence, which can be found on the Resource tab. For more information, see Viewing Resources.
You can use the pin
tag to pin a specific input version as shown below below:
Example
resources: - name: <string> type: DistributionRule configuration: pin: versionId: <number>
The following resources support version pinning:
Output Resources
Output resources are resources that are either generated or changed by a step. When specified as the output of a step, the resource receives the output of the step. If required, this output resource can then be used as an input resource in a subsequent step in the same pipeline or another pipeline. The output resource can also be referred to by its name as an argument in the shell commands that the step executes.
Output Resource Definiton
A resource can be specified as an output for a step by adding it in the outputResources
section of a step.
YAML Schema
steps: - name: <step_name> type: <step_type> configuration: outputResources: - name: <resource name> branch: <string> # see description of defaults below
Tag | Description of usage | Required/Optional |
---|---|---|
| Name of the declared resource that is to be used as an input for the step. | Required |
| A | Optional |
Example - Output Resource Definition
steps: - name: step_2 type: Bash configuration: outputResources: - name: my_repo branch: master
Viewing Resources in the UI
After a pipeline's YAML file is committed to a repository, add the repository to Pipelines through the UI. The Pipelines platform then watches for changes (job additions, edits or deletes) through source control webhooks. YAML changes are automatically synced and are reflected in the UI immediately.
After the pipeline source successfully syncs the YAML file, select Applications | My Pipelines to view the the pipeline.
In the Pipelines view:
Each resource is shown as a circular icon
Clicking a resource displays information specific to that resource
Clicking the YAML
icon → Resources tab displays the
resources
definition for the pipeline