Matrix

JFrog Pipelines Documentation

ft:sourceType
Paligo

The Matrix step commences multiple parallel build processes across multiple containers, with different settings for each.

Common use cases for Matrix steps are:

  • Splitting a large test suite into smaller units, and executing them in parallel to reduce total execution time.

  • Testing against multiple values of environment variables or multiple runtime images.

  • Testing against multiple base operating system versions.

The matrix configuration can specify:

  • Multiple sets of environment variable definitions

  • Multiple runtime images

  • Multiple node pools.

  • Multiple operating systems

The Matrix step executes the specified shell scripts multiple times in parallel steplets, in each specified runtime for each set of environment variables on each specified platform. For example, if a Matrix step specifies 3 sets of environment variables and 2 runtime images, it will run a total of 6 steplets (3 times in runtime 1, 3 times in runtime 2). If the Matrix step also specifies 2 node pools, it will run 12 steplets.

A Matrix step can be optionally preceded by a PreMatrix step to prepare the build node environment for the steplets, and a PostMatrix step to aggregate information produced by the steplets. For more information, see Using the Matrix Step.

Note

Matrix step is available for Linux nodes only.

Usage

Matrix

pipelines:
  - name: <string>
    steps:
      - name: <string>
        type: Matrix
        stepMode: Bash
        configuration: 
          #inherits from Bash 
          multiNode: <boolean>        # optional, only needed if steplets 
                                      # need to execute on separate nodes

        execution:
          onStart:
            - echo "Preparing for work..."
          onExecute:                            # required     
            - echo "Executing steplet $step_name"
            - echo "env1 = $env1"
            - echo "env2 = $env2"
          onSuccess:
            - echo "Job well done!"
          onFailure:
            - echo "uh oh, something went wrong"
          onComplete: #always
            - echo "Cleaning up some stuff"

        stepletMultipliers:
           nodePools:                             # wants to execute step against       
            - windows                             # multiple operating systems
            - ubuntu_18
            - ubuntu_16
 
          environmentVariables:       # optional, only needed if user
            - env1: one               # wants to execute step against
              env2: two               # multiple values of env
            - env1: abc                
              env2: xyz

          runtimes:                   # optional, only needed if user
            - type: image                         # wants to execute step against 
                          image:                  # multiple images
                auto:                 
                  language:
                  version:
                custom:
                  <same config as Bash>
              image:               
                auto:
                  language:
                  version:
                custom:
                  <same config as Bash>

          fastFail: <true/false>     # default false. If specified, matrix
                                     # step fails when any steplet fails, 
                                     # unless specified in allowFailures

          allowFailures:             # optional, array of combinations
            - nodePool:              # that are allowed to fail  
              environmentVariables:
                env1: abc                              
                env2: xyz
              runtime:               

          exclude:                   # optional, array of combinations
            - nodePool:              # that are not executed  
              environmentVariables:
                env1: one
                env2: two
              runtime:  
Tags
name

An alphanumeric string (underscores are permitted) that identifies the step.

type

Must be Matrix for this step type.

stepmode

Specifies the runtime OS mode. May be either Bash or PowerShell. If not specified, defaults to Bash.

configuration

Specifies all configuration selections for the step's execution environment. This step inherits the Bash/ PowerShell step configuration tags, including these pertinent tags:

Tag

Description of usage

Required/Optional

inputStep

May specify a PreMatrix preamble step.

Optional

In addition, these tags can be defined to support the step's native operation:

Tag

Description of usage

Required/Optional

multiNode

When true, steplets will be required to run on separate nodes. Default is false.

For more information, see Multi-node Matrix.

Optional

stepletMultipliers

In addition, these tags can be defined to support the step's native operation:

Tag

Description of usage

Required/Optional

nodePools

A collection of node pool names. Recognized only when multiNode is set to true. When specified, the step will be executed in each of the defined runtimes for each set of environmentVariables on each node pool.

If not defined, the step will be executed in the default node pool, or the node pool specified in the configuration section.

Example

steps:
- name: step_1
  type: Matrix
  stepMode: Bash
  configuration:
    multiNode: true
  stepletMultipliers:
    environmentVariables:
      - foo: foo
      - bar: bar
    nodePools:
      - gcp
      - aws 

Optional

environmentVariables

A collection of sets of environment variable definitions. Each set of definitions will be used in an execution of a steplet in each of the defined runtimes.

If not defined, then a single steplet will execute for each of the defined runtimes on each of the specified nodePools.

Optional

runtimes

A collection of runtime definitions. The step will be executed in each defined runtime, on each of the specified nodePools, for each set of environmentVariables.

If not defined, then each steplet will be executed in the default runtime, or the runtime specified in the configuration section.

Optional

fastFail

When set to true the step fails when any steplet fails, unless permitted to fail in allowFailures.

Default is false.

Optional

allowFailures

A collection that specifies the combinations of nodepools, environmentVariables, and runtimes that are permitted to fail without terminating the entire Matrix step. This only has effect when fastFail is set to true.

Optional

exclude

A collection that specifies the combinations of nodepools, environmentVariables, and runtimes that will not be executed.

Optional

execution

Declares collections of shell command sequences to perform for pre- and post-execution phases:

Tag

Description of usage

Required/Optional

onStart

Commands to execute in advance of onExecute

Optional

onExecute

Main commands to execute for the step

Optional

onSuccess

Commands to execute on successful completion

Optional

onFailure

Commands to execute on failed completion

Optional

onComplete

Commands to execute on any completion

Optional

Caution

onExecuteonStartonSuccessonFailure, and onComplete are reserved keywords. Using these keywords in any other context in your execution scripts can cause unexpected behavior.

Examples
Example 1

This is an example of how to use the Matrix step to perform a build activity.

Matrix Step Example

      - name: matrix_example
        type: Matrix
        stepMode: Bash
        configuration:
          inputSteps: 
            - name: matrix_fan_out_example

        stepletMultipliers:
          environmentVariables:             # Sets of environment variables for steplets
            - animal:    dog                            # - Set 1
              mineral:   copper
              vegetable: carrot
            - animal:    goat                           # - Set 2
              mineral:   iron
              vegetable: broccoli
            - animal:    lizard             # - Set 3
              mineral:   lead        

          nodePools:
                - gcp
                - aws 
          
                  runtimes:                         # Runtimes for steplets
            - type: image
              image:
                auto:
                  language: java
                  versions:
                    - 13.0
            - type: image
              image:
                auto:
                  language: node
                  versions:
                    - 8.17.0
        execution:
          onExecute:
            - echo "I am executing matrix steplet ${steplet_id}"
            - echo "Envs animal= $animal , mineral = $mineral , vegetable = $vegetable"
            - echo "Runtime image name: $step_image_name | Runtime image tag: $step_image_tag"

When run, the above example will execute in a total of 6 steplets: in 2 runtimes for each of 3 sets of environment variables:

Runtime

Set 1

Set 2

Set 3

java 13.0

dog, copper, carrrot

goat, iron, broccoli

lizard, lead, <null>

node 8.17.0

dog, copper, carrrot

goat, iron, broccoli

lizard, lead, <null>

Example 2 - allowFailures, Exclude, fastFail
pipelines:
  - name: S_Matrix_0052
    steps:           
      - name: S_Matrix_0052
        type: Matrix
        stepMode: Bash
        stepletMultipliers:
          fastFail: true
          allowFailures:
            - environmentVariables:
                delay: 1
                fail: 'true'
              nodePool: windows
              runtime: host
            - environmentVariables:
                delay: 1
                fail: 'true'
              nodePool: ubuntu
              runtime: host
          exclude:
            - environmentVariables:
                delay: 1
                fail: 'true'
              nodePool: windows
              runtime: host
           - environmentVariables:
                delay: 1
                fail: 'true'
              nodePool: ubuntu
              runtime: host 
          environmentVariables:
            - delay: 1
              fail: 'true'
            - delay: 2
              fail: 'false'
            - delay: 3
              fail: 'false'
            - delay: 5
              fail: 'false' 
        execution:
          onStart:
            - sleep "$delay"
          onExecute:
            - echo "I am on matrix step"
            - |
              [ $fail == 'false' ] 
            - ls -l .