To enable retention policy at the system-level, in the system.yaml
file, in the core section, set the retentionPolicy’s enabled
flag to true
. This policy applies to all your pipelines.
Note
By default, Retention Policy is disabled at system-level, which effectively means infinite retention.
When enabled, system-level Retention Policy uses two properties to control which pipelines run data is to be retained:
Max Age Days: Specifies the number of days, after which the pipelines run data will be deleted. This can be set in granularity of days. Default value is 0 (infinite retention).
Minimum Runs: Specifies the minimum number of pipeline runs to keep, regardless of their age. Default value is 10.
The following optional cron expression properties helps you control when the Retention Policy is triggered:
Cron Hours: Cron expression interval in hours when the Retention Policy runs. Default value is 4.
Cron Days: Cron expression days of the week when the Retention Policy runs. Input should be comma separated names or numbers 0-7 (0 or 7 are Sunday). Default value is 6 (Saturday).
Cron Time Zone: Cron time zone that defines the time zone when Retention Policy runs. List of supported time zones. Default value is UTC.
Example
Configure retention policy in system.yaml
core: retentionPolicy: enabled: true maxAgeDays: 90 minRuns: 10 cronHours: 4 cronDays: "0,6" cronTimeZone: "UTC"
In this example, system-level Retention Policy is enabled, and pipelines run data for the last 90 days is set to be retained and data older than 90 days is set to be deleted. The minimum runs property (minRuns
) ensures that for each one of your pipelines, the latest 10 pipeline runs data is retained, regardless of the policy’s max allowed age (maxAgeDays
). This means, if some (or all) of your 10 latest pipelines runs are older than 90 days, Pipelines will still keep their run data. If you have less than 10 pipeline runs for certain pipelines, Pipelines will keep all their pipelines run data. As for cron properties, Retention Policy will be triggered every 4 hours on Saturday (6) and Sunday(0) and will start at midnight UTC.