PKS tile updates fail on applying add-ons

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

·         Applying changes from OpsMan fails. One or more clusters show to be in the failed state when looked at from the pks cli "pks clusters". The details of the cluster "pks cluster <cluster-name>" shows the details of the failure as follows:
Last Action Description:  Instance update failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: a8913acc-c314-4b75-a625-45e7e05a7deb, broker-request-id: 72a3fe58-9490-45f5-832f-41b43e38da44, task-id: 6359, operation: update, error-message: 0 succeeded, 1 errored, 0 canceled

·         Reviewing the bosh logs from the cli "bosh task <id> --debug" show messages similar to the following:
failed to start all system specs after 1200 with exit code 1\n","stderr":"Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nThe StorageClass \"thindisk\" is invalid: parameters: Forbidden: updates to parameters are forbidden.

Environment

VMware PKS 1.x

Cause

The PKS tile plan that the cluster is using has an apply add-on that is attempting to make a change to the storage class parameters. This is not allowed as storage classes are immutable objects.

Resolution

Verify the problem storage class exists by listing the storage classes and find the one matching in the error message using "kubectl get storageclass --all-namespaces". For example "thindisk" is the name of the storage class in the message "kubectl apply\nThe StorageClass \"thindisk\" is invalid: parameters: Forbidden: updates to parameters are forbidden". 

Review the current parameters of the storage class. For example "kubectl get storageclass thindisk -o yaml"

Compare the current parameters with the parameters from the PKS plan apply add-ons from the OpsMan. Looks for the section starting with "apiVersion: storage.k8s.io/v1" and/or "kind: StorageClass".

Determine the reason the parameters where changes. If they were changes in the apply add-ons after the cluster was created, change them back and use a new plan for the new apply add-ons. If another action deleted and recreated the storage class with different parameters, like with helm charts. Instead have that action use a different storage class name. Then recreate the effected storage class back to match what is in the PKS plan apply add-ons section. 

Workaround:

In the apply add-ons section of the cluster plan remove the storage class section starting with "apiVersion: storage.k8s.io/v1" and/or "kind: StorageClass". This will have the cluster errands no longer attempt to create or change the storage class.