ArgoCD Finalizer Shield: Protecting your clusters from unintended deletion

ArgoCD is a popular tool by the CNCF (Cloud Native Computing Foundation) for managing and deploying applications on Kubernetes clusters. It provides a declarative and GitOps-based approach to application delivery, allowing DevOps teams to automate and standardize their workflows across different environments. However, as the number of clusters and applications grows, so does the complexity of managing them. 

In ArgoCD, clusters are represented as Kubernetes Secrets stored in the ArgoCD namespace. It’s important to note that these Secrets can be renamed or deleted when changing a cluster’s name for example. This can have significant consequences if done accidentally or without proper precautions in place. Therefore, it’s crucial to have adequate safeguards; for example, enable auditing or logging to monitor changes made to the Secrets, so that you can quickly identify any unauthorized modifications and take action to address them in place to prevent unintentional or unauthorized modification of these Secrets.

In this blog post, we’ll explore how to protect your ArgoCD clusters from accidental deletion using finalizers, a simple yet powerful mechanism that ensures the integrity of your cloud-native infrastructure

Cluster Representation in ArgoCD

Managed clusters are stored within Secrets in the ArgoCD namespace. The ApplicationSet controller uses those same Secrets to generate parameters to identify and target available clusters.

Here’s an example of how a secret stores the authentication details and other configuration settings of a cluster:

apiVersion: v1
kind: Secret
data:
  config: 
  name: 
  server: 
metadata:
  annotations:
    argocd.argoproj.io/refresh: "2023-04-16T15:06:58Z"
    managed-by: argocd.argoproj.io
    meta.helm.sh/release-name: argocd-prod-use1
    meta.helm.sh/release-namespace: argocd-prod-use1
  creationTimestamp: "2022-06-21T07:57:05Z"
  labels:
    app.kubernetes.io/instance: argocd-prod-use1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
    argocd.argoproj.io/secret-type: cluster
    cloud_provider: aws
    env: production
    helm.sh/chart: argo-cd-5.27.2
    is_dr: "false"
    region: prod-2-usw2
  name: argocd-cluster-aws-k8s-prod-2-usw2
  namespace: argocd-prod-use1
type: Opaque

How Changing the Name of an ArgoCD Cluster Can Lead to Issues

When you rename a cluster in ArgoCD, the old cluster and all of its associated resources are deleted and replaced with a new cluster with the altered name. This can cause discrepancies in projects where resources were not automatically synchronized, as the deleted objects will not be recreated automatically, resulting in potential incidents.

Uncovering the Potential of the .syncPolicy.preserveResourcesOnDeletion Option in ArgoCD to Protect Resources

ArgoCD offers a level of control over resource modification with the preserveResourcesOnDeletion option, which can be used to prevent associated resources from being deleted when the application is deleted. To enable this setting, set .syncPolicy.preserveResourcesOnDeletion (put in italics) to true in the ApplicationSet. While this can be a useful way to protect resources from accidental deletion, it may not be enough to protect all resources in all situations. If all applications are not tagged with this option, there may still be out-of-sync resources that are not protected. To ensure comprehensive protection, automation should be created to continuously check the flag of all ApplicationSets and alert owners of any discrepancies. This may require additional effort to implement, but it’s necessary to ensure all resources are adequately protected.

What are Kubernetes Finalizers and how can we use them as Shield?

Kubernetes Finalizers are a powerful mechanism that enables objects to perform necessary cleanup or other actions before they are deleted. They are essentially hooks that prevent an object from being deleted until all dependent resources have been deleted first, ensuring that all resources associated with the object are properly removed. This helps to maintain the integrity of the system and prevent errors from occurring.

Many Kubernetes objects are automatically assigned the ‘kubernetes’ finalizer by default, which prevents the Kubernetes API server from deleting an object until all of its associated resources have been properly cleaned up. This ensures that all related resources are handled correctly before the object is removed. To accomplish this, we have decided to add finalizers to all clusters managed by ArgoCD. As part of any ArgoCD deployment or upgrade, all secrets that are labeled with argocd.argoproj.io/secret-type=cluster are automatically patched (kubectl patch) with the following YAML:

apiVersion: v1
kind: Secret
metadata:
  finalizers:
  - kubernetes

With this configuration in place, ArgoCD can’t delete any cluster without removing the Finalizer from the metadata.finalizers field. This configuration ensures you won’t accidentally delete a cluster and cause an incident, like for example not syncing a Global Network Policy critical rule that should be synced to your cluster.

If you have decided to remove outdated and unnecessary clusters, carefully review all resources created by the cluster in ArgoCD (e.g., filter according to cluster). Make sure that none of these resources are still needed and won’t be affected if deleted. Once you are sure, you can safely delete the finalizer and the cluster.

Secure Your Cloud-Native Infrastructure with ArgoCD Finalizer

In summary, implementing finalizers in your ArgoCD deployment is a straightforward and effective way to prevent inadvertent cluster deletions, and to ensure that your cloud-native infrastructure runs smoothly. We encourage you to give it a try and share your results with us!