Managing Pipeline Sources

JFrog Pipelines Documentation
ft:sourceType
Paligo

Overview

A Pipeline Source represents a source control repository (such as GitHub or BitBucket) where Pipelines definition files can be found. A pipeline source connects to the repository through an integration.After a pipeline source is added, Pipelines automatically loads all config files from the repository that matches the specified filename filter.

Note

Administrator users can create, view, edit and sync pipeline sources. Non-administrator users can only view and sync pipeline sources.

Administering Pipeline Sources

Only users with administratorprivileges can manage pipeline sources.Introduction to Users and Groups

Requirements

To successfully add a pipeline source, ensure the following:

Adding a Pipeline Source (1.31.0 and higher)

To add a source control repository as a Pipeline Source:

  1. Go to Administration | Pipelines | Pipeline Sources.

  2. In the resulting Pipeline Sources display, clickAdd Pipeline Source. and click From YAML.

  3. In the Add YAML Pipeline Source page, click one of the following:

    • Single Branch

    • Multi Branch

  4. Click to select the protocol type to be used for cloning the repository when the pipeline source is synced:

    • SSH

    • HTTPS

  5. Complete the Single/Multi Branch form:

    Field

    Description

    Name

    Enter a unique name for the pipeline source.

    SCM Provider Integration

    Click the SCM Provider Integration field and select your source control integration from the dropdown list. Only integrations that are compatible for use as a Pipeline Source will be included in the dropdown list.

    Repository Full Name

    Based on the information you have provided for the selected integration, such as API token, all the relevant repositories are listed in the Repository Full Name field. Select the path of the repository where your pipeline files are stored. If the name of the repository is not auto-fetched, enter the full name of the repository where your pipeline files are stored.

    Branch (Single Branch only)

    Based on the SCM provider and Repository Full Name you have provided, all the available branches are automatically fetched (for example, main). Select the required branch. If the name of the branch is not auto-fetched, enter the name of the branch.

    Exclude (Multi Branch only)

    Specify the Exclude Branch Patternas a regular expression for the matching branch names to exclude.

    Include (Multi Branch only)

    Specify the Include Branch Pattern as a regular expression for the matching branch names to include.

    Sync all branches toggle

    (Multi Branch only)

    When a new multi branch pipeline source is added, only the default branched is synced. To sync all the branches based on the Exclude/Include pattern, click the Sync all branches toggle. If Exclude/Include pattern is not provided, all branches are synced.

    In addition, when a new commit is pushed to thematching branch, that particular branch is synced.

    Folder Name

    Provide the name of the directory where the YAML config is available.

    • To use Folder Name, in the SCM repository, place your YAML files in a directory named .jfrog-pipelines. This directory can be the root or one level below the root. Directories beyond this level are not supported.

    • Following are the possible directory structures for monorepos:

      - root

      ------ .jfrog-pipelines

      Or

      - root

      ---- .jfrog-pipelines

      -------- service1

      -------- service2

      Or

      - root

      ---- service1

      --------- .jfrog-pipelines

      ---- service2

      --------- .jfrog-pipelines

    • The .jfrog-pipelinesdirectory can contain any number of YAML files.

    • To parse all YAML files in the root directory, enter ".". However, other pipeline sources cannot point to this repository.

    • You can create multiple pipeline sources pointing to the same SCM repository as long as the directory names are different (and none of them point to root).

    Recommended Directory Structure for using Folder Name

    Here are some examples of the supported directory structure:

    YAML Files Location

    Folder Name Path

    YAML files are in the .jfrog-pipelines root directory

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch all the YAML files

    YAML files are in the .jfrog-pipelines/project1 directory

    Enter.jfrog-pipelines/project1in theFolder Namefield to fetch the YAML files in the.jfrog-pipelines/project1directory

    YAML files are in the following directories:

    • .jfrog-pipelines

    • .jfrog-pipelines/project1

    • .jfrog-pipelines/project2

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch all the YAML files in all directories

    or

    Enter .jfrog-pipelines/project1 in the Folder Name field to fetch the YAML files in the .jfrog-pipelines/project1 directory

    Enter .jfrog-pipelines/project2 in the Folder Name field to fetch the YAML files in the .jfrog-pipelines/project2 directory

    YAML files are in the following directories:

    • .jfrog-pipelines

    • service1/.jfrog-pipelines

    • service2/.jfrog-pipelines

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch the YAML files in the .jfrog-pipelines directory, but YAML files in the service1 /.jfrog-pipelines and service2 /.jfrog-pipelines directories will not be included.

    or

    Enter service1 /.jfrog-pipelines in the Folder Name field to fetch specific YAML files in the service1 /.jfrog-pipelines directory

    Enter service2 /.jfrog-pipelines in the Folder Name field to fetch specific YAML files in the service2 /.jfrog-pipelines directory

    If you have a monorepo with multiple services within a single repo, a directory structure such as the following is recommended:

    Existing Directory Structure

    Recommended Directory Structure

    Root

    Build/ci

    -Service1

    -Build/ci/pipe.yaml

    -Service2

    -Build/ci/pipe.yaml

    Root

    .jfrog-pipelines

    -Service1/pipe.yaml

    -Service2/pipe.yaml

Adding a Pipeline Source (1.30.0 and lower)

To add a source control repository as a Pipeline Source:

  1. Go to Administration | Pipelines | Pipeline Sources.

  2. In the resulting Pipeline Sources display, clickAdd Pipeline Source and click one of the following:

    • From YAML: Add the pipeline source from a YAML configuration file, which contains the declarations of all resources and workflow steps.

    • From Template: Add the pipeline source from a pre-defined template by providing a values file. For more information, see Pipelines Templates.

      add_pipeline_source_1.18_aug162021.png
From YAML

  1. In the Add YAML Pipeline Source page, click one of the following:

    • Single Branch

    • Multi Branch

  2. Click to select the protocol type to be used for cloning the repository when the pipeline source is synced:

    • SSH

    • HTTPS

  3. Complete the Single/Multi Branch form:

    Field

    Description

    Name

    Enter a unique name for the pipeline source.

    SCM Provider Integration

    Click the SCM Provider Integration field and select your source control integration from the dropdown list. Only integrations that are compatible for use as a Pipeline Source will be included in the dropdown list.

    Repository Full Name

    Based on the information you have provided for the selected integration, such as API token, all the relevant repositories are listed in the Repository Full Name field. Select the path of the repository where your pipeline files are stored. If the name of the repository is not auto-fetched, enter the full name of the repository where your pipeline files are stored.

    Branch (Single Branch only)

    Based on the SCM provider and Repository Full Name you have provided, all the available branches are automatically fetched (for example, main). Select the required branch. If the name of the branch is not auto-fetched, enter the name of the branch.

    Exclude (Multi Branch only)

    Specify the Exclude Branch Patternas a regular expression for the matching branch names to exclude.

    Include (Multi Branch only)

    Specify the Include Branch Pattern as a regular expression for the matching branch names to include.

    Sync all branches toggle

    (Multi Branch only)

    When a new multi branch pipeline source is added, only the default branched is synced. To sync all the branches based on the Exclude/Include pattern, click the Sync all branches toggle. If Exclude/Include pattern is not provided, all branches are synced.

    In addition, when a new commit is pushed to thematching branch, that particular branch is synced.

    Folder Name (Pipelines 1.27.0 and higher)

    Provide the name of the directory where the YAML config is available.

    • To use Folder Name, in the SCM repository, place your YAML files in a directory named .jfrog-pipelines. This directory can be the root or one level below the root. Directories beyond this level are not supported.

    • Following are the possible directory structures for monorepos:

      - root

      ------ .jfrog-pipelines

      Or

      - root

      ---- .jfrog-pipelines

      -------- service1

      -------- service2

      Or

      - root

      ---- service1

      --------- .jfrog-pipelines

      ---- service2

      --------- .jfrog-pipelines

    • The .jfrog-pipelinesdirectory can contain any number of YAML files.

    • To parse all YAML files in the root directory, enter ".". However, other pipeline sources cannot point to this repository.

    • You can create multiple pipeline sources pointing to the same SCM repository as long as the directory names are different (and none of them point to root).

    Recommended Directory Structure for using Folder Name

    Here are some examples of the supported directory structure:

    YAML Files Location

    Folder Name Path

    YAML files are in the .jfrog-pipelines root directory

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch all the YAML files

    YAML files are in the .jfrog-pipelines/project1 directory

    Enter.jfrog-pipelines/project1in theFolder Namefield to fetch the YAML files in the.jfrog-pipelines/project1directory

    YAML files are in the following directories:

    • .jfrog-pipelines

    • .jfrog-pipelines/project1

    • .jfrog-pipelines/project2

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch all the YAML files in all directories

    or

    Enter .jfrog-pipelines/project1 in the Folder Name field to fetch the YAML files in the .jfrog-pipelines/project1 directory

    Enter .jfrog-pipelines/project2 in the Folder Name field to fetch the YAML files in the .jfrog-pipelines/project2 directory

    YAML files are in the following directories:

    • .jfrog-pipelines

    • service1/.jfrog-pipelines

    • service2/.jfrog-pipelines

    Enter . (dot) or .jfrog-pipelines in theFolder Namefield to fetch the YAML files in the .jfrog-pipelines directory, but YAML files in the service1 /.jfrog-pipelines and service2 /.jfrog-pipelines directories will not be included.

    or

    Enter service1 /.jfrog-pipelines in the Folder Name field to fetch specific YAML files in the service1 /.jfrog-pipelines directory

    Enter service2 /.jfrog-pipelines in the Folder Name field to fetch specific YAML files in the service2 /.jfrog-pipelines directory

    If you have a monorepo with multiple services within a single repo, a directory structure such as the following is recommended:

    Existing Directory Structure

    Recommended Directory Structure

    Root

    Build/ci

    -Service1

    -Build/ci/pipe.yaml

    -Service2

    -Build/ci/pipe.yaml

    Root

    .jfrog-pipelines

    -Service1/pipe.yaml

    -Service2/pipe.yaml

    Migrating from Config File Filter to the New Folder Structure

    In Pipelines version 1.27.0, Config File Filter has been replaced with a new field called Folder Name for adding your pipeline sources. Though Config File Filter is still supported and available while editing a pipeline source.

    • This change does not affect pipeline sources added before 1.26.0. However, it is highly recommended that you use the recommended folder structure for new pipeline sources and move your existing sources to the recommended folder structure. Using the folder structure significantly improves the performance of your pipeline sync.

    • You can take advantage of the Folder Name feature by editing the pipeline source and then providing the new directory path. In this case, the entry in the Config File Filter is ignored and the pipeline source in the directory path is used.

    • If both the fields have values, then the Folder Namefield takes precedence. However, if no.jfrog-pipelinesdirectory is found in the directory path, then information in theConfig File Filterfield is used to find the YAML files.

    Config File Filter (Pipelines 1.26.0 and lower)

    Enter a regular expression. Any filenames that match the expression will be loaded.

    Examples:

      • If you plan to declare your resources in a file named pipelines.resources.yml and your pipelines in pipelines.workflows.yml, then you might enter pipelines.*.yml.

      • If you have multiple yaml config files, then enter .*yml.

      • In addition to the pipelines.yml, if you are using a values.yml, ensure both files are added to the filter, using (pipelines|values).yml.

From Template

  1. Complete the Template Propertiesform:

    select_template_1.png

    • Click the Select Template Namespace field and select the namespace.

    • Click the Select Template Names field and select the template.

    • Click the Select Template Version field and select the version for the template.

  2. Click Next.

    In the resulting Specify values filepage, click one of the following from theRead YAML values from drop-down:

    • Source Control Repository: Use this when you want to fetch the values.yml file from an SCM provider.

    • Enter Manually: Use this when you want to manually the define the values for the pipeline source.

  3. Source Control RepositoryClick one of the following:

    • Single Branch

    • Multi Branch

  4. Click to select the protocol type to be used for cloning the repository when the pipeline source is synced:

    • SSH

    • HTTPS

  5. Complete theSingle/Multi Branch form:

    Field

    Description

    Name

    Enter a unique name for the pipeline source.

    SCM Provider Integration

    Click the SCM Provider Integration field and select your source control integration from the dropdown list. Only integrations that are compatible for use as a Pipeline Source will be included in the dropdown list.

    Repository Full Name

    Based on the information you have provided for the selected integration, such as API token, all the relevant repositories are listed in the Repository Full Name field. Select the path of the repository where your pipeline files are stored. If the name of the repository is not auto-fetched, enter the full name of the repository where your pipeline files are stored.

    Branch (Single Branch only)

    Based on the SCM provider and Repository Full Name you have provided, all the available branches are automatically fetched (for example, main). Select the required branch. If the name of the branch is not auto-fetched, enter the name of the branch.

    Exclude (Multi Branch only)

    Specify the Exclude Branch Patternas a regular expression for the matching branch names to exclude.

    Include (Multi Branch only)

    Specify the Include Branch Pattern as a regular expression for the matching branch names to include.

    Sync all branches toggle

    (Multi Branch only)

    When a new multi branch pipeline source is added, only the default branched is synced. To sync all the branches based on the Exclude/Include pattern, click the Sync all branches toggle. If Exclude/Include pattern is not provided, all branches are synced.

    In addition, when a new commit is pushed to thematching branch, that particular branch is synced.

    Values File Path

    Enter the path to the .values.yml file, which contains the values for the template.

  6. Enter Manually

    • Name: Enter a unique name to identify a pipeline source definition created from the template.

    • Values.yml: Manually enter or paste the Values definition.

  7. Click Create Source to complete adding the Pipeline Source.

If JFrog Pipelines successfully syncs with the repository, the Statusfield displays "Success". Otherwise, it displays "Failed".

pipeline_source_1.18_aug162021.png

All pipeline config files committed to the repository will be automatically loaded into JFrog Pipelines, and all defined pipelines from that source are listed in the pipeline list.

Validating YAML

Before adding a pipeline source, you have the option of validating your pipelines, resources, and values YAMLs and then committing it to the SCM. This enables you to get instant feedback on pipeline sync errors. The YAML validator will validate your YAML for both semantic and syntactic errors.

To validate your YAML:

  1. Click Validate YAML.

  2. Either enter the YAML and paste the YAML contents in the Pipelines YAML and Resources YAML fields. If you have a values YAML, enter the YAML contents in the Values YAML field. For more information about the Pipelines DSL, see Defining a Pipeline.

    As the YAML content is entered, it is validated and syntactic errors, if any, are displayed.

  3. After fixing all the syntactic errors, click Validate to validate the YAML for semantic errors.

180127356.png
Viewing Pipeline Sources

To view the list of pipeline sources already added to Pipelines, go to Administration | Pipelines | Pipeline Sources or Application | Pipelines | Pipeline Sources.

The page displays the list of pipelines sources that are available to you according to the permission targets defined in the JFrog Platform. Your user account must be granted permissionsfor a pipeline source for it to be shown.Introduction to Permissions

Each row of the pipeline sources list includes the following:

Property

Description

Name

  • For new pipeline sources: This is the name you provided while creating the pipeline source

  • For existing pipeline sources: This defaults to repositoryName/branch for single branch and repositoryName for multi-branch

Git Repository

The source repository path of the pipeline source

Project

The Project that the pipeline source belongs to

Latest Status

The success/failure status of the last sync

Config File Filter

The filter string for the pipeline config files

Last Sync

The time and date of the last sync

Changed By

Name of the user who made the last update to the pipeline source

Context

The commit SHA that triggered the last sync

Logs

Click the Logs link to view the log from the last sync. Use this to diagnose a failure to sync a pipeline source.

If the pipeline source is a multi branch source, the row presents aggregate information for all branches, and can be expanded/collapsed to show the sync status of each branch.

Syncing a Pipeline Source

When any of the pipeline config files have changed, you will need to sync the pipeline source to reload:

  1. Go to Administration | Pipelines | Pipeline Sources and click the Actions button located at the far right.

  2. Click Sync.

pipeline_source_sync_aug172021.png
Viewing Sync Logs

To view the logs of all synchronized config files:

  1. Go to Administration | Pipelines | Pipeline Sources and click the Actions button located at the far right.

  2. Click Logs.

pipeline_source_logs_aug172021.png

Pipeline Source Sync Recovery

Pipelines provides an option to enable auto-sync for pipelines sources that are out of sync. For more information, see Pipeline Source Sync Recovery.

Deleing a Pipeline Source

When a pipeline source is deleted, it deletes all the pipelines associated with the source.

To delete a pipeline source:

  1. Go to Administration | Pipelines | Pipeline Sources and click the Actions button located at the far right.

  2. Click Delete.