The Matrix step commences multiple parallel build processes across multiple containers, with different settings for each.
Common use cases for Matrix steps are:
Splitting a large test suite into smaller units, and executing them in parallel to reduce total execution time.
Testing against multiple values of environment variables or multiple runtime images.
Testing against multiple base operating system versions.
The matrix configuration can specify:
Multiple sets of environment variable definitions
Multiple runtime images
Multiple node pools
Multiple operating systems
The Matrix step executes the specified shell scripts multiple times in parallel steplets, in each specified runtime for each set of environment variables on each specified platform. For example, if a Matrix step specifies 3 sets of environment variables and 2 runtime images, it will run a total of 6 steplets (3 times in runtime 1, 3 times in runtime 2). If the Matrix step also specifies 2 node pools, it will run 12 steplets.
A Matrix step can be optionally preceded by a PreMatrix step to prepare the build node environment for the steplets, and a PostMatrix step to aggregate information produced by the steplets. For more information, see Using the Matrix Step.
Note
Matrix step is available for Linux nodes only.
Usage
Matrix
pipelines: - name: <string> steps: - name: <string> type: Matrix stepMode: Bash configuration: #inherits from Bash concurrency: <string> multiNode: <boolean> # optional, only needed if steplets # need to execute on separate nodes execution: onStart: - echo "Preparing for work..." onExecute: # required - echo "Executing steplet $step_name" - echo "env1 = $env1" - echo "env2 = $env2" onSuccess: - echo "Job well done!" onFailure: - echo "uh oh, something went wrong" onComplete: # always - echo "Cleaning up some stuff" stepletMultipliers: nodePools: # wants to execute step against - windows # multiple operating systems - ubuntu_18 - ubuntu_16 environmentVariables: # optional, only needed if user - env1: one # wants to execute step against env2: two # multiple values of env - env1: abc env2: xyz runtimes: # optional, only needed if user - type: image # wants to execute step against image: # multiple images auto: language: version: custom: <same config as Bash> image: auto: language: version: custom: <same config as Bash> fastFail: <true/false> # default false. If specified, matrix # step fails when any steplet fails, # unless specified in allowFailures allowFailures: # optional, array of combinations - nodePool: # that are allowed to fail environmentVariables: env1: abc env2: xyz runtime: exclude: # optional, array of combinations - nodePool: # that are not executed environmentVariables: env1: one env2: two runtime:
Tags
name
An alphanumeric string (underscores are permitted) that identifies the step.
type
Must be Matrix
for this step type.
stepmode
Specifies the runtime OS mode. May be either Bash
or PowerShell
. If not specified, defaults to Bash
.
configuration
Specifies all configuration selections for the step's execution environment. This step inherits the Bash/ PowerShell step configuration tags, including these pertinent tags:
Tag | Description of usage | Required/Optional |
---|---|---|
| May specify a PreMatrix preamble step. | Optional |
In addition, these tags can be defined to support the step's native operation:
Tag | Description of usage | Required/Optional |
---|---|---|
| When For more information, see Multi-node Matrix. NoteWhen both | Optional |
| Define the number of steplets that can concurrently run on a node. This helps in optimizing resource usage and ensures that the node efficiently handles steplets without being overwhelmed. NoteWhen both | Optional |
stepletMultipliers
In addition, these tags can be defined to support the step's native operation:
Tag | Description of usage | Required/Optional |
---|---|---|
| A collection of node pool names. Recognized only when If not defined, the step will be executed in the default node pool, or the node pool specified in the Example steps: - name: step_1 type: Matrix stepMode: Bash configuration: multiNode: true stepletMultipliers: environmentVariables: - foo: foo - bar: bar nodePools: - gcp - aws | Optional |
| A collection of sets of environment variable definitions. Each set of definitions will be used in an execution of a steplet in each of the defined If not defined, then a single steplet will execute for each of the defined | Optional |
| A collection of runtime definitions. The step will be executed in each defined runtime, on each of the specified If not defined, then each steplet will be executed in the default runtime, or the runtime specified in the | Optional |
| When set to Default is false. | Optional |
| A collection that specifies the combinations of | Optional |
| A collection that specifies the combinations of | Optional |
execution
Declares collections of shell command sequences to perform for pre- and post-execution phases:
Tag | Description of usage | Required/Optional |
---|---|---|
| Commands to execute in advance of | Optional |
| Main commands to execute for the step | Optional |
| Commands to execute on successful completion | Optional |
| Commands to execute on failed completion | Optional |
| Commands to execute on any completion | Optional |
Caution
onExecute
, onStart
, onSuccess
, onFailure
, and onComplete
are reserved keywords. Using these keywords in any other context in your execution scripts can cause unexpected behavior.
Examples
Example 1
This is an example of how to use the Matrix step to perform a build activity.
Matrix Step Example
- name: matrix_example type: Matrix stepMode: Bash configuration: inputSteps: - name: matrix_fan_out_example stepletMultipliers: environmentVariables: # Sets of environment variables for steplets - animal: dog # - Set 1 mineral: copper vegetable: carrot - animal: goat # - Set 2 mineral: iron vegetable: broccoli - animal: lizard # - Set 3 mineral: lead nodePools: - gcp - aws runtimes: # Runtimes for steplets - type: image image: auto: language: java versions: - 13.0 - type: image image: auto: language: node versions: - 8.17.0 execution: onExecute: - echo "I am executing matrix steplet ${steplet_id}" - echo "Envs animal= $animal , mineral = $mineral , vegetable = $vegetable" - echo "Runtime image name: $step_image_name | Runtime image tag: $step_image_tag"
When run, the above example will execute in a total of 6 steplets: in 2 runtimes for each of 3 sets of environment variables:
Runtime | Set 1 | Set 2 | Set 3 |
---|---|---|---|
java 13.0 | dog, copper, carrrot | goat, iron, broccoli | lizard, lead, <null> |
node 8.17.0 | dog, copper, carrrot | goat, iron, broccoli | lizard, lead, <null> |
Example 2 - allowFailures, Exclude, fastFail
pipelines: - name: S_Matrix_0052 steps: - name: S_Matrix_0052 type: Matrix stepMode: Bash stepletMultipliers: fastFail: true allowFailures: - environmentVariables: delay: 1 fail: 'true' nodePool: windows runtime: host - environmentVariables: delay: 1 fail: 'true' nodePool: ubuntu runtime: host exclude: - environmentVariables: delay: 1 fail: 'true' nodePool: windows runtime: host - environmentVariables: delay: 1 fail: 'true' nodePool: ubuntu runtime: host environmentVariables: - delay: 1 fail: 'true' - delay: 2 fail: 'false' - delay: 3 fail: 'false' - delay: 5 fail: 'false' execution: onStart: - sleep "$delay" onExecute: - echo "I am on matrix step" - | [ $fail == 'false' ] - ls -l .
Example 3 - concurrency
pipelines: - name: matrix_check steps: - name: step1 type: Matrix stepMode: Bash configuration: concurrency: 50 stepletMultipliers: environmentVariables: # Sets of environment variables for steplets - animal: dog # - Set 1 mineral: copper vegetable: carrot - animal: goat # - Set 2 mineral: iron vegetable: broccoli - animal: lizar1d # - Set 3 mineral: lea1d - animal: liza2rd # - Set 3 mineral: le2ad - animal: liza3rd # - Set 3 mineral: lea3d - animal: liza4rd # - Set 3 mineral: lea4d - animal: liza5rd # - Set 3 mineral: le5ad - animal: liz6ard # - Set 3 mineral: lea6d - animal: lizar7d # - Set 3 mineral: le7ad - animal: lizar8d # - Set 3 mineral: lea8d - animal: liza9rd # - Set 3 mineral: lea9d - animal: liza10rd # - Set 3 mineral: lead10 - animal: liza11rd # - Set 3 mineral: lead11 - animal: liza12rd # - Set 3 mineral: lead12 - animal: lizar13d # - Set 3 mineral: lead13 - animal: d1og14 # - Set 1 mineral: copper14 vegetable: carrot14 - animal: go1at15 # - Set 2 mineral: ir1on15 vegetable: broccoli15 - animal: lizard16 # - Set 3 mineral: lead16 - animal: lizard17 # - Set 3 mineral: lead17 - animal: lizard18 # - Set 3 mineral: lead18 - animal: lizard19 # - Set 3 mineral: lead20 - animal: lizard20 # - Set 3 mineral: lead20 - animal: lizard21 # - Set 3 mineral: lead21 - animal: lizard22 # - Set 3 mineral: lead22 - animal: lizard23 # - Set 3 mineral: lead23 - animal: lizard24 # - Set 3 mineral: lead24 - animal: lizard25 # - Set 3 mineral: lead25 - animal: lizard26 # - Set 3 mineral: lead26 - animal: lizard27 # - Set 3 mineral: lead27 - animal: lizard28 # - Set 3 mineral: lead28 - animal: dog29 # - Set 1 mineral: copper29 vegetable: carrot29 - animal: goat30 # - Set 2 mineral: iron30 vegetable: broccoli30 - animal: lizard31 # - Set 3 mineral: lead31 - animal: lizard32 # - Set 3 mineral: lead32 - animal: lizard33 # - Set 3 mineral: lead33 - animal: lizard34 # - Set 3 mineral: lead34 - animal: lizard35 # - Set 3 mineral: lead35 - animal: lizard36 # - Set 3 mineral: lead36 - animal: lizard37 # - Set 3 mineral: lead37 - animal: lizard38 # - Set 3 mineral: lead38 - animal: lizard39 # - Set 3 mineral: lead39 - animal: lizard40 # - Set 3 mineral: lead40 - animal: lizard41 # - Set 3 mineral: lead41 - animal: lizard42 # - Set 3 mineral: lead42 - animal: lizard43 # - Set 3 mineral: lead43 - animal: dog44 # - Set 1 mineral: copper44 vegetable: carrot44 - animal: goat45 # - Set 2 mineral: iron45 vegetable: broccoli45 - animal: lizard46 # - Set 3 mineral: lead46 - animal: lizard47 # - Set 3 mineral: lead47 - animal: lizard48 # - Set 3 mineral: lead48 - animal: lizard49 # - Set 3 mineral: lead49 - animal: lizard50 # - Set 3 mineral: lead50 execution: onExecute: - echo "I am executing matrix steplet ${steplet_id}" - echo "Envs animal= $animal , mineral = $mineral , vegetable = $vegetable