The Matrix step commences multiple parallel build processes across multiple containers, with different settings for each.
Common use cases for Matrix steps are:
Splitting a large test suite into smaller units, and executing them in parallel to reduce total execution time.
Testing against multiple values of environment variables or multiple runtime images.
Testing against multiple base operating system versions.
The matrix configuration can specify:
Multiple sets of environment variable definitions
Multiple runtime images
Multiple node pools
Multiple operating systems
The Matrix step executes the specified shell scripts multiple times in parallel steplets, in each specified runtime for each set of environment variables on each specified platform. For example, if a Matrix step specifies 3 sets of environment variables and 2 runtime images, it will run a total of 6 steplets (3 times in runtime 1, 3 times in runtime 2). If the Matrix step also specifies 2 node pools, it will run 12 steplets.
A Matrix step can be optionally preceded by a PreMatrix step to prepare the build node environment for the steplets, and a PostMatrix step to aggregate information produced by the steplets. For more information, see Using the Matrix Step.
Note
Matrix step is available for Linux nodes only.
Usage
Matrix
pipelines:
- name: <string>
steps:
- name: <string>
type: Matrix
stepMode: Bash
configuration:
#inherits from Bash
concurrency: <string>
multiNode: <boolean> # optional, only needed if steplets
# need to execute on separate nodes
execution:
onStart:
- echo "Preparing for work..."
onExecute: # required
- echo "Executing steplet $step_name"
- echo "env1 = $env1"
- echo "env2 = $env2"
onSuccess:
- echo "Job well done!"
onFailure:
- echo "uh oh, something went wrong"
onComplete: # always
- echo "Cleaning up some stuff"
stepletMultipliers:
nodePools: # wants to execute step against
- windows # multiple operating systems
- ubuntu_18
- ubuntu_16
environmentVariables: # optional, only needed if user
- env1: one # wants to execute step against
env2: two # multiple values of env
- env1: abc
env2: xyz
runtimes: # optional, only needed if user
- type: image # wants to execute step against
image: # multiple images
auto:
language:
version:
custom:
<same config as Bash>
image:
auto:
language:
version:
custom:
<same config as Bash>
fastFail: <true/false> # default false. If specified, matrix
# step fails when any steplet fails,
# unless specified in allowFailures
allowFailures: # optional, array of combinations
- nodePool: # that are allowed to fail
environmentVariables:
env1: abc
env2: xyz
runtime:
exclude: # optional, array of combinations
- nodePool: # that are not executed
environmentVariables:
env1: one
env2: two
runtime:
Tags
name
An alphanumeric string (underscores are permitted) that identifies the step.
type
Must be Matrix for this step type.
stepmode
Specifies the runtime OS mode. May be either Bash or PowerShell. If not specified, defaults to Bash.
configuration
Specifies all configuration selections for the step's execution environment. This step inherits the Bash/ PowerShell step configuration tags, including these pertinent tags:
Tag | Description of usage | Required/Optional |
|---|---|---|
| May specify a PreMatrix preamble step. | Optional |
In addition, these tags can be defined to support the step's native operation:
Tag | Description of usage | Required/Optional |
|---|---|---|
| When For more information, see Multi-node Matrix. NoteWhen both | Optional |
| Define the number of steplets that can concurrently run on a node. This helps in optimizing resource usage and ensures that the node efficiently handles steplets without being overwhelmed. NoteWhen both | Optional |
stepletMultipliers
In addition, these tags can be defined to support the step's native operation:
Tag | Description of usage | Required/Optional |
|---|---|---|
| A collection of node pool names. Recognized only when If not defined, the step will be executed in the default node pool, or the node pool specified in the Example steps:
- name: step_1
type: Matrix
stepMode: Bash
configuration:
multiNode: true
stepletMultipliers:
environmentVariables:
- foo: foo
- bar: bar
nodePools:
- gcp
- aws | Optional |
| A collection of sets of environment variable definitions. Each set of definitions will be used in an execution of a steplet in each of the defined If not defined, then a single steplet will execute for each of the defined | Optional |
| A collection of runtime definitions. The step will be executed in each defined runtime, on each of the specified If not defined, then each steplet will be executed in the default runtime, or the runtime specified in the | Optional |
| When set to Default is false. | Optional |
| A collection that specifies the combinations of | Optional |
| A collection that specifies the combinations of | Optional |
execution
Declares collections of shell command sequences to perform for pre- and post-execution phases:
Tag | Description of usage | Required/Optional |
|---|---|---|
| Commands to execute in advance of | Optional |
| Main commands to execute for the step | Optional |
| Commands to execute on successful completion | Optional |
| Commands to execute on failed completion | Optional |
| Commands to execute on any completion | Optional |
Caution
onExecute, onStart, onSuccess, onFailure, and onComplete are reserved keywords. Using these keywords in any other context in your execution scripts can cause unexpected behavior.
Examples
Example 1
This is an example of how to use the Matrix step to perform a build activity.
Matrix Step Example
- name: matrix_example
type: Matrix
stepMode: Bash
configuration:
inputSteps:
- name: matrix_fan_out_example
stepletMultipliers:
environmentVariables: # Sets of environment variables for steplets
- animal: dog # - Set 1
mineral: copper
vegetable: carrot
- animal: goat # - Set 2
mineral: iron
vegetable: broccoli
- animal: lizard # - Set 3
mineral: lead
nodePools:
- gcp
- aws
runtimes: # Runtimes for steplets
- type: image
image:
auto:
language: java
versions:
- 13.0
- type: image
image:
auto:
language: node
versions:
- 8.17.0
execution:
onExecute:
- echo "I am executing matrix steplet ${steplet_id}"
- echo "Envs animal= $animal , mineral = $mineral , vegetable = $vegetable"
- echo "Runtime image name: $step_image_name | Runtime image tag: $step_image_tag"
When run, the above example will execute in a total of 6 steplets: in 2 runtimes for each of 3 sets of environment variables:
Runtime | Set 1 | Set 2 | Set 3 |
|---|---|---|---|
java 13.0 | dog, copper, carrrot | goat, iron, broccoli | lizard, lead, <null> |
node 8.17.0 | dog, copper, carrrot | goat, iron, broccoli | lizard, lead, <null> |
Example 2 - allowFailures, Exclude, fastFail
pipelines:
- name: S_Matrix_0052
steps:
- name: S_Matrix_0052
type: Matrix
stepMode: Bash
stepletMultipliers:
fastFail: true
allowFailures:
- environmentVariables:
delay: 1
fail: 'true'
nodePool: windows
runtime: host
- environmentVariables:
delay: 1
fail: 'true'
nodePool: ubuntu
runtime: host
exclude:
- environmentVariables:
delay: 1
fail: 'true'
nodePool: windows
runtime: host
- environmentVariables:
delay: 1
fail: 'true'
nodePool: ubuntu
runtime: host
environmentVariables:
- delay: 1
fail: 'true'
- delay: 2
fail: 'false'
- delay: 3
fail: 'false'
- delay: 5
fail: 'false'
execution:
onStart:
- sleep "$delay"
onExecute:
- echo "I am on matrix step"
- |
[ $fail == 'false' ]
- ls -l .Example 3 - concurrency
pipelines:
- name: matrix_check
steps:
- name: step1
type: Matrix
stepMode: Bash
configuration:
concurrency: 50
stepletMultipliers:
environmentVariables: # Sets of environment variables for steplets
- animal: dog # - Set 1
mineral: copper
vegetable: carrot
- animal: goat # - Set 2
mineral: iron
vegetable: broccoli
- animal: lizar1d # - Set 3
mineral: lea1d
- animal: liza2rd # - Set 3
mineral: le2ad
- animal: liza3rd # - Set 3
mineral: lea3d
- animal: liza4rd # - Set 3
mineral: lea4d
- animal: liza5rd # - Set 3
mineral: le5ad
- animal: liz6ard # - Set 3
mineral: lea6d
- animal: lizar7d # - Set 3
mineral: le7ad
- animal: lizar8d # - Set 3
mineral: lea8d
- animal: liza9rd # - Set 3
mineral: lea9d
- animal: liza10rd # - Set 3
mineral: lead10
- animal: liza11rd # - Set 3
mineral: lead11
- animal: liza12rd # - Set 3
mineral: lead12
- animal: lizar13d # - Set 3
mineral: lead13
- animal: d1og14 # - Set 1
mineral: copper14
vegetable: carrot14
- animal: go1at15 # - Set 2
mineral: ir1on15
vegetable: broccoli15
- animal: lizard16 # - Set 3
mineral: lead16
- animal: lizard17 # - Set 3
mineral: lead17
- animal: lizard18 # - Set 3
mineral: lead18
- animal: lizard19 # - Set 3
mineral: lead20
- animal: lizard20 # - Set 3
mineral: lead20
- animal: lizard21 # - Set 3
mineral: lead21
- animal: lizard22 # - Set 3
mineral: lead22
- animal: lizard23 # - Set 3
mineral: lead23
- animal: lizard24 # - Set 3
mineral: lead24
- animal: lizard25 # - Set 3
mineral: lead25
- animal: lizard26 # - Set 3
mineral: lead26
- animal: lizard27 # - Set 3
mineral: lead27
- animal: lizard28 # - Set 3
mineral: lead28
- animal: dog29 # - Set 1
mineral: copper29
vegetable: carrot29
- animal: goat30 # - Set 2
mineral: iron30
vegetable: broccoli30
- animal: lizard31 # - Set 3
mineral: lead31
- animal: lizard32 # - Set 3
mineral: lead32
- animal: lizard33 # - Set 3
mineral: lead33
- animal: lizard34 # - Set 3
mineral: lead34
- animal: lizard35 # - Set 3
mineral: lead35
- animal: lizard36 # - Set 3
mineral: lead36
- animal: lizard37 # - Set 3
mineral: lead37
- animal: lizard38 # - Set 3
mineral: lead38
- animal: lizard39 # - Set 3
mineral: lead39
- animal: lizard40 # - Set 3
mineral: lead40
- animal: lizard41 # - Set 3
mineral: lead41
- animal: lizard42 # - Set 3
mineral: lead42
- animal: lizard43 # - Set 3
mineral: lead43
- animal: dog44 # - Set 1
mineral: copper44
vegetable: carrot44
- animal: goat45 # - Set 2
mineral: iron45
vegetable: broccoli45
- animal: lizard46 # - Set 3
mineral: lead46
- animal: lizard47 # - Set 3
mineral: lead47
- animal: lizard48 # - Set 3
mineral: lead48
- animal: lizard49 # - Set 3
mineral: lead49
- animal: lizard50 # - Set 3
mineral: lead50
execution:
onExecute:
- echo "I am executing matrix steplet ${steplet_id}"
- echo "Envs animal= $animal , mineral = $mineral , vegetable = $vegetable