Error "panic: interface conversion: interface {} is nil" when creating a TKGm Workload Cluster with Node Pools' variables.overrides field

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Creating a TKGm Workload Cluster with nodePools' variables.overrides fields fails with below error:

$ tanzu cluster create --file <cluster>.yaml
panic: interface conversion: interface {} is nil, not stringgoroutine 1 [running]:
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectArrayInterfaceType({0xc000a26080, 0x4, 0xc000600000?}, {0xc000aa40a0, 0x4d}, 0x46115c?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:410 +0x81a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x2561e80?, 0xc000284228}, {0xc000aa40a0, 0x4d}, 0x433c55?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:346 +0x13a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectArrayInterfaceType({0xc00041e430, 0x1, 0xc000600000?}, {0xc0002de1c0, 0x3e}, 0x46115c?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:431 +0x64b
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x2561e80?, 0xc000284240}, {0xc0002de1c0, 0x3e}, 0xc000bd48c0?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:346 +0x13a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x27012e0?, 0xc0002a4000}, {0xc0002de180, 0x34}, 0xc000bd4a18?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:364 +0x64a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x27012e0?, 0xc000da05d0}, {0xc0002aa0f0, 0x2a}, 0x433c55?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:364 +0x64a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectArrayInterfaceType({0xc000a26100, 0x3, 0xc000600000?}, {0xc0002aa0c0, 0x28}, 0x46115c?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:400 +0x2c9
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x2561e80?, 0xc000284288}, {0xc0002aa0c0, 0x28}, 0xc000bd4dc8?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:346 +0x13a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x27012e0?, 0xc000da05a0}, {0xc0002a8060, 0x15}, 0xc000bd4f20?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:364 +0x64a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x27012e0?, 0xc000e02180}, {0xc0002a2210, 0xd}, 0x433c55?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:364 +0x64a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.processYamlObjectAndAddToMap({0x27012e0?, 0xc000e020c0}, {0x2ce8c56, 0x4}, 0x430cf5?)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:364 +0x64a
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.(*tkgctl).processClusterObjectForConfigurationVariables(0x7ffce99cd6c3?, {0xc000e02000}, {0x7ffce99cd6c3, 0x3f})
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/helpers.go:274 +0x1d4
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.(*tkgctl).processWorkloadClusterInputFile(0xc0001c24b0, 0xc000bd5958, 0x0)
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/create_cluster.go:199 +0x14c
github.com/vmware-tanzu/tanzu-framework/tkg/tkgctl.(*tkgctl).CreateCluster(_, {0x0, 0x1, 0x0, {0x7ffce99cd6c3, 0x3f}, {0x0, 0x0}, {0x0, 0x0}, ...})
        /go/src/cayman-tkg/tkg/src/tkg/tkgctl/create_cluster.go:62 +0x7b
main.createCluster({0x0, 0x0}, 0xc0004b50e0)
        /go/src/cayman-tkg/tkg/src/cmd/cli/plugin/cluster/create.go:167 +0x358
main.create(0x4bc03e0?, {0xc0004f4330, 0x0, 0x3?})
        /go/src/cayman-tkg/tkg/src/cmd/cli/plugin/cluster/create.go:124 +0xfc
github.com/spf13/cobra.(*Command).execute(0x4bc03e0, {0xc0004f42d0, 0x3, 0x3})
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0x974
github.com/spf13/cobra.(*Command).ExecuteC(0xc000005800)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x40c
github.com/spf13/cobra.(*Command).Execute(0xc000005800?)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:1039 +0x13
github.com/vmware-tanzu/tanzu-plugin-runtime/plugin.(*Plugin).Execute(0xc000a84b08)
        /go/pkg/mod/github.com/vmware-tanzu/[email protected]/plugin/plugin.go:46 +0x2b
main.main()
        /go/src/cayman-tkg/tkg/src/cmd/cli/plugin/cluster/main.go:56 +0x1c8

 

The ClusterClass-based cluster's YAML file includes nodePools' variables.overrides as follows:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
spec:
[...]
  topology:
    class: tkg-vsphere-default-2.5.0
[...]
    workers:
      machineDeployments:
      - class: tkg-worker
[...]
        variables:
          overrides:
          - name: nodePoolLabels
            value:
            - key: <key_1>
              value: <value_1>

Environment

TKGm prior to 2.5.3

Cause

Known bug impacting TKG versions prior to 2.5.3.

Fix is included in TKG 2.5.3.

Resolution

Generally, the only supported way to create a TKGm Workload Cluster is via "tanzu cluster create" command. Due to this known bug, the command would fail when nodePools' variables.overrides fields are included in the cluster's YAML file.

The workaround in this scenario is to use "kubectl apply -f <cluster>.yaml" command to create the Workload Cluster. This command shouldn't throw the mentioned error and the cluster can be created successfully.

Note: for all other scenarios, please continue using "tanzu cluster create" command to create your Workload Clusters.

Additional Information

If after creating the cluster you realize the associated worker nodes are missing the overridden configuration, please refer to KB https://knowledge.broadcom.com/external/article?articleNumber=389785