Choosing Node Pools

JFrog Pipelines Documentation

Products
JFrog Pipelines
Content Type
User Guide
ft:sourceType
Paligo

A pipeline can, if necessary, control through Pipelines DSL which nodes it or any individual step will execute in.

When an administrator user sets up node pools in Pipelines, one of the node pools is designated as the default. This is the set of nodes that all pipeline steps will execute in when no explicit node pool is specified by the pipeline or step.

If you need to run your pipeline steps in a specific node pool, you can specify this in your pipeline's DSL. You can also configure any step to run on a node from a specific node pool.

You may wish to specify a node pool when:

  • Your pipeline or step is resource-intensive, so you need to execute in a specific node pool that uses more powerful machines

  • You need to run the pipeline or step on a specific OS (e.g., Ubuntu or CentOS)

  • You need to run a pipeline or step in a specific node pool for security (for example, behind a firewall in your own data center)

Note

Steps that are specified to run in the same affinityGroup must always run in the same node pool.

Node Pools

Your Pipelines configuration must have at least one node pool, with at least one node, for pipelines to execute. These must be set up by an administrator user, with one node pool set as the default. A typical configuration will have been set up with multiple node pools to provide different choices for operating system and computing platforms.

For example, an administrator user may have set up these node pools and made them available to you for executing your pipelines:

  1. A static node pool of Ubuntu 20.04 nodes running self-hosted, set as the default node pool

  2. A dynamic node pool of Ubuntu 20.04 nodes running on Amazon Web Services

  3. A static node pool of CentOS 7 nodes running self-hosted

You can view the node pools available to you from the Pipelines Node Pools tab:

nodePools_07mar22.png

Example Pipeline

In your pipeline's DSL you can designate the pipeline or any individual steps to run in a node pool by its name.

  • Pipeline node pool: You can specify a specific node pool for the pipeline in the configuration block of the pipeline, using the nodePool tag. When specified here, all steps in the pipeline will execute in nodes from that node pool.

  • Step node pool: You can specify a specific node pool for any step by declaring the nodePool tag in the step's configuration block. This will override any node pool designation for the pipeline.

The following example pipeline specifies that all steps of the pipeline will execute in nodes from the dynamic node pool ubuntu20aws on Amazon Web Services. The second step, however, overrides this as it is configured to run on an self-hosted node from the static node pool ubuntu20.

pipelines.yml

pipelines:
  - name: select_nodepool_example
    configuration:
      nodePool:   ubuntu20aws                       # Run in the dynamic node pool on AWS
    steps:
      - name: nodePool_step1
        type: Bash
        execution:
          onExecute:
            - echo "$step_name is running in $step_platform on $step_node_pool_name"
 
      - name: nodePool_step2
        type: Bash
        configuration:
          nodePool:   ubuntu20                       # Run in the static node pool self-hosted
          inputSteps:
            - name: nodePool_step1
        execution:
          onExecute:
            - echo "$step_name is running in $step_platform on $step_node_pool_name"

      - name: nodePool_step3
        type: Bash
        configuration:
          inputSteps:
            - name: nodePool_step1
        execution:
          onExecute:
            - echo "$step_name is running in $step_platform on $step_node_pool_name"

When the pipeline is run:

  • Steps nodePool_step1 and nodePool_step3 will execute on nodes in ubuntu20aws

  • Step nodePool_step2 will execute on a node in ubuntu20