When using vSphere Kubernetes Service 3.4, cannot use VMClass with precreated disk controllers
Article ID: 449993
Updated On:
Products
Issue/Introduction
If an environment where a VMClass has disk controllers defined, new machines in a VKS cluster may fail to provision.
This is typically done for GPU workloads where multiple disk controllers are attached to work around device hot add not being available in PCI passthrough / vGPU.
For example:
This VMClass explicitly adds 3 separate ParaVirtualSCSIController devices, one each on busNumber: 0, 1, and 2:
- device:
_typeName: ParaVirtualSCSIController
busNumber: 0
...
- device:
_typeName: ParaVirtualSCSIController
busNumber: 1
...
- device:
_typeName: ParaVirtualSCSIController
busNumber: 2
...kubectl get ma -A
NAMESPACE NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION
example example-cluster-00000-00001 example-cluster example-CPVM1 vsphere://00000000-XXXX Running 23h v1.33.3+vmware.1-fips
example example-cluster-00000-00002 example-cluster example-CPVM2 vsphere://00000000-XXXX Running 24h v1.33.3+vmware.1-fips
example example-cluster-00000-00003 example-cluster example-CPVM3 vsphere://00000000-XXXX Running 24h v1.33.3+vmware.1-fips
example example-workers1-XXXX example-cluster example-workers1-XXXX vsphere://00000000-XXXX Running 4d21h v1.32.7+vmware.3-fips
example example-cluster-workers1-YYYY example-cluster vsphere://00000000-XXXX Provisioned 19m v1.33.3+vmware.1-fips
When describing the node, it reported "Waiting for a Node with spec.providerID"
Message: * NodeHealthy: Waiting for a Node with spec.providerID vsphere://00000000-XXXX to exist
Observed Generation: 3
Reason: ReadyUnknown
Status: Unknown
Type: Ready
Last Transition Time: ####-##-##T##:####Z
Message:
Observed Generation: 3
Reason: UpToDate
Status: True
Type: UpToDate
Last Transition Time: ####-##-##T##:####Z
Message:
Observed Generation: 3
Reason: Ready
Status: True
Type: BootstrapConfigReady
Last Transition Time: ####-##-##T##:####Z
Message:
Observed Generation: 3
Reason: Ready
Status: True
Type: InfrastructureReady
Last Transition Time: ####-##-##T##:####Z
Message: Waiting for a Node with spec.providerID vsphere://00000000-XXXX to exist
Observed Generation: 3
Reason: InspectionFailed
Status: Unknown
Type: NodeHealthy
Last Transition Time: ####-##-##T##:####Z
Message: Waiting for a Node with spec.providerID vsphere://00000000-XXXX to exist
Observed Generation: 3
Reason: InspectionFailed
Status: Unknown
Type: NodeReady
Last Transition Time: ####-##-##T##:####Z
Message:
Observed Generation: 3
Reason: NotPaused
Status: False
Type: Paused
Last Transition Time: ####-##-##T##:####Z
Message:
Observed Generation: 3
Reason: NotDeleting
Status: False
Type: Deleting
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal DetectedUnhealthy 4m8s (x44 over 19m) machinehealthcheck-controller Machine example-workers1-XXXX has unhealthy Node
The worker node's cloud init log indicates a failure to mount disk, when looking at /var/lib/cloud-init-output.log:
{"time":"####-##-##T##:####-Z","level":"INFO","msg":"start mount","disk":"/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:1:0"}
{"time":"####-##-##T##:####Z","level":"ERROR","msg":"error applying: error applying task mount-var-lib-containerd: error mounting disk to [/var/lib/containerd]: error getting block devices for [/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:1:0]: error obtaining block device info for [/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:1:0]: exit status 32"}
Environment
VMware vSphere Kubernetes Service v3.4 and above
Cause
VKS logic was added to have strict error handling when disks were miss mounted to avoid unintended data loss, however the use case of multiple preallocated SCSI controllers was not considered.
Resolution
Broadcom Engineering is aware of this issue. A future release of the vSphere Kubernetes Service (VKS) will support multiple preallocated SCSI controllers. Additionally, VMware Cloud Foundation (VCF) will extend the maximum number of disks that can be allocated per PVSCSI controller in VKS to 64.
Workaround:
To resolve this issue, apply the following two-part workaround on the vCenter Server Appliance (VCSA) and the affected VM Class.
Step 1: Enable Support for 64 Disks per PVSCSI Controller in VM Service
Connect to the vCenter Server Appliance via SSH and log in with root credentials.
Open the following configuration file using a text editor (such as vi): /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml
Locate the <pvscsiCtrlr256DiskSupportEnabled> parameter and change its value to true.
Save and close the file.
Restart the vsanmgmtd service for the changes to take effect by running the following command:
service-control --restart vmware-vsan-health
NOTE: This configuration change is not persistent across updates. You must reapply these steps after every vCenter Server upgrade.
Step 2: Remove Controllers from the VM Class:
Navigate to the target VM Class and edit its configuration.
Remove all PVSCSI controllers from the VM Class.
Verify that the total number of disks attached to the VKS node (specifically when using PCI Passthrough or a vGPU) is less than 64.
Save the configuration. The rollout of the changes will proceed automatically.
Feedback
