When steps in a pipeline must execute on the same build node, they can be assigned to the same affinity group. This may be necessary when a step requires access to data produced by a prior step that remains in the build node's runtime environment.
Because Pipelines distributes the execution of steps across nodes in a node pool, no step can necessarily rely on files or artifacts produced by a prior step being available, since the earlier step may have executed in a different build node. While the preferred practice is often to create a stateful pipeline, you may need to bind steps to the same build node instead through an affinity group. The path of a directory available to all steps in the group is provided as an environment variable named
shared_workspace and may be used to share files between steps.
Of course, if the node pool used by your pipeline only has a single node, then your steps may still succeed without binding them into an affinity group (since all steps execute on the same single node). However, since an administrator can increase the number of nodes in any node pool at any time, you should still use affinity groups to bind steps that need them, to ensure that they continue to execute successfully when conditions change.
Steps in the same affinity group cannot run in parallel.
When defining steps in affinity groups, ensure that there are no intermediate steps that belong to a different affinity group. Doing this will cause the pipeline to fail.
For example, when defining affinity groups for three steps, do one of the following:
Define Step A, Step B, and Step C in the same affinity group
Define Step A and Step B in the same affinity group, with Step C in a different affinity group
Define Step A in an affinity group, with Step B and Step C in a different affinity group
Usage of affinityGroup Tag
Steps are bound to an affinity group through its
configuration section, using the
To specify an affinity group for an individual step:
steps: - name: <name> type: <step_type> configuration: affinityGroup: <any string>
You can specify any string as the name of the affinity group, but you must use the same string for all
affinityGroup tags in the steps you wish to bind together in the group.
Alternatively, if all steps in the pipeline are to be in the same affinity group,
affinityGroup may be specified in the Pipelines configuration section:
pipelines: - name: <name> configuration: affinityGroup: <any string> steps: - <step>
The following simplified pipeline creates two steps bound to the same
pipelines: - name: pipeline_affinityGroup_example steps: - name: ag_step_1 type: Bash configuration: affinityGroup: together execution: onExecute: - echo "Running " $pipeline_name "|" $step_name > $shared_workspace/myoutput.txt - cat $shared_workspace/myoutput.txt - name: ag_step_2 type: Bash configuration: affinityGroup: together inputSteps: - name: ag_step_1 execution: onExecute: - cat $shared_workspace/myoutput.txt
ag_step_1 creates a text file in the build node where it executes, using the
shared_workspace variable to ensure that the file will be available to both steps. The subsequent step,
ag_step_2 tries to read the file.
ag_step_2 into the same affinity group, the file that
ag_step_1 created in the build node's file system is available to
ag_step_2 because it is guaranteed to be running in the same node. Without being bound to an affinity group,
ag_step_2 may fail because it might execute in a node where the file does not exist.
When this pipeline is loaded in the JFrog Platform, the interactive diagram shown in the Pipeline History indicates that these steps are bound into the same affinity group:
When the pipeline is run, the Pipeline Run Logs show that
ag_step_2 was able to successfully find the file in the node's filesystem that was created by