Description
Kubernetes statefulset has many properties and most are not changeable after deployment. When running a helm upgrade, helm will generate a new statefulset template. If the new template changed anything other than 'replicas', 'template', and 'updateStrategy' in the spec filed, you will see the following messages:
Error: UPGRADE FAILED: StatefulSet.apps "artifactory-ha-artifactory-ha-member" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden && StatefulSet.apps "artifactory-ha-artifactory-ha-primary" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
In some cases, the error message will give the file that's rejecting the change. However, for a generic error like the one above, you will need to find out what is causing the immutable error.
Changes in the statefulset template during helm upgrade usually come from two sources: changes in values.yaml and changes in chart version ( which changes the statefulset template in chart ). Here are some tips to the debug steps:
Debug and Fix
1. Identify the values.yaml change is usually the first step. You may be passing a new value with upgrade, or didn't pass a value that was previous set this time. Use the command below to get the current effective values and compare.
$ helm get values <release-name> -n namespace
$ helm get values <release-name> -n namespace -A
Namespace is not needed on helm v2 and -A will give you all the current values including chart default values
2. If you're certain the change didn't come from values.yaml, check the template directly.
It may be due to changes in the StatefulSet template itself - often caused by a chart version update, local modifications, or updates to ConfigMaps/Secrets.
Confirm whether the chart version changed during the upgrade (specify the version in the upgrade command if possible).
Check for any local modifications (e.g., if the chart was installed from a local path).
Look at the age of related ConfigMaps/Secrets—if they were recently updated, the StatefulSet might be referencing them.
Finally, run the upgrade command with --dry-run to generate the updated StatefulSet template, and compare it to the current one.
$ helm upgrade ..... --dry-run
Note:
Make sure to check out the upgrade_notes of each product and search for any message related to "upgrade" in the change log. For example, you may notice the following in changelogs.
https://github.com/jfrog/charts/blob/3b75f96ec9df3bc89c4c5e86c8df17e8d0e80aea/stable/distribution/CHANGELOG.md#10281---june-22-2021
If this is an upgrade and you are using the default PostgreSQL (postgresql.enabled=true), you need to pass the previous 9.x/10.x/12.x's postgresql.image.tag, previous postgresql.persistence.size and databaseUpgradeReady=true
If you didn't pass postgresql.image.tag values, helm upgrade may pick up different values from upgrade.
Thus, it will create a new postgresql statefulset template which will result in the same upgrade forbidden error because you could be changing an forbidden field.
3. Helm client versions or kubernetes versions - the upgrade forbidden error could also be a result of the helm client version or Kubernetes versions.
In such a case, you will not be able to find the difference in steps 1-3.
Check if there has been any change to the helm client or Kubernetes version.
4. Fix - there are a few ways to work around the issue if we are not able to identify the source of the change:
You can always delete the statefulset before running the upgrade command. Statefulset is a controller; deleting it does not have any direct impact. Just make sure you don't also delete the pod until upgrade command creates a new statefulset.
After new statefulset is created, you may need to delete the pod for new pod to be generated ( Usually ,k8s automatically creates new pod itself ) Another easier option is to use the --force command with upgrade which is in some sort similar to the above.
Keep in mind, however, that there is currently a bug with k8s (https://github.com/kubernetes/kubernetes/issues/91459) which will lead to the following error if you use the --force option:
Error: UPGRADE FAILED: failed to replace object: Service "artifactory-ha" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "artifactory-ha-primary" is invalid: spec.clusterIP: Invalid value: "": field is immutable