Setting Retention Policy

Retention Policy enables you to configure a policy to discard pipelines run data from older pipelines runs. Pipelines run data includes Pipeline Run Logs, Pipeline State, Steps Cache, and Runs State. This data, over time and depending on your Pipelines usage, can become very large and lead to increased storage costs.

Retention Policy can be set at two levels:

System-level
Pipeline-level

Setting Retention Policy at System-level

To enable retention policy at the system-level, in the system.yaml file, in the core section, set the retentionPolicy’s enabled flag to true. This policy applies to all your pipelines.

Note

By default, Retention Policy is disabled at system-level, which effectively means infinite retention.

When enabled, system-level Retention Policy uses two properties to control which pipelines run data is to be retained:

Max Age Days: Specifies the number of days, after which the pipelines run data will be deleted. This can be set in granularity of days. Default value is 0 (infinite retention).
Minimum Runs: Specifies the minimum number of pipeline runs to keep, regardless of their age. Default value is 10.

The following optional cron expression properties helps you control when the Retention Policy is triggered:

Cron Hours: Cron expression interval in hours when the Retention Policy runs. Default value is 4.
Cron Days: Cron expression days of the week when the Retention Policy runs. Input should be comma separated names or numbers 0-7 (0 or 7 are Sunday). Default value is 6 (Saturday).
Cron Time Zone: Cron time zone that defines the time zone when Retention Policy runs. List of supported time zones. Default value is UTC.

Example

Configuring retention policy in system.yaml

core: 
    retentionPolicy:
        enabled: true
        maxAgeDays: 90
        minRuns: 10
        cronHours: 4
        cronDays: "0,6"
        cronTimeZone: "UTC"

In this example, system-level Retention Policy is enabled, and pipelines run data for the last 90 days is set to be retained and data older than 90 days is set to be deleted. The minimum runs property (minRuns) ensures that for each one of your pipelines, the latest 10 pipeline runs data is retained, regardless of the policy’s max allowed age (maxAgeDays). This means, if some (or all) of your 10 latest pipelines runs are older than 90 days, Pipelines will still keep their run data. If you have less than 10 pipeline runs for certain pipelines, Pipelines will keep all their pipelines run data. As for cron properties, Retention Policy will be triggered every 4 hours on Saturday (6) and Sunday(0) and will start at midnight UTC.

Setting Retention Policy at Pipeline-level

Once Retention Policy is enabled at system-level, it can also be configured at at pipeline-level. For information about setting the values for pipeline-level Retention Policy, see Defining a Pipeline.

Just like Retention Policy at system-level, you can control the values of the number of days and minimum number of runs to keep at pipeline-level. These settings override the system-level Retention Policy, but pipeline-level policy values can only beequal to or lesser than the system-level configuration.

For example, if the system-level Retention Policy configuration is:

 retentionPolicy:
    enabled: true
    maxAgeDays: 90
    minRuns: 10

And the pipeline-level Retention Policy configuration is:

 retentionPolicy:
    maxAgeDays: 30
    minRuns: 5

Then Pipelines will discard Pipeline run data for runs that are older than 30 days. Pipelines, however, will also keep at least 5 pipeline runs data, even if they are older than 30 days (or even if they are older than system-level 90 days value).

Note

If you set the values of pipeline-level Retention Policy to greater than your system-level values, the Pipeline Source Sync will fail.

Example

pipelines:
 - name: example_pipeline_retention_policy
   configuration:
     retentionPolicy:
       maxAgeDays: 30
       minRuns: 5
   steps:
     - name: step_1
       type: Bash          
       execution:
         onExecute:
           - echo "Hello world"