Supervisor Cluster Upgrade Stalls with "ServerFaultCode: A general system error occurred: vix error codes = (1, 0)."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The upgrade of the Supervisor Cluster does not complete due to the following error.

A general system error occurred: vix error codes = (1, 0)

• In /var/log/vmware/wcp/wcpsvc.log, error messages like the following are repeatedly output, and the configuration of the Supervisor Control Plane VM does not complete even after several hours.
  
  YYYY-MM-DDTHH:MM:SS.###Z error wcp [vclib/guestop_manager.go:54] [opID=########-########-####-####-####-############-vm-<ID>] Guest operation failed with err ServerFaultCode: A general system error occurred: vix error codes = (1, 0).

YYYY-MM-DDTHH:MM:SS.###Z info wcp [vclib/guestop_manager.go:57] [opID=########-########-####-####-####-############-vm-<ID>] Guest operation failure is not retriable.
YYYY-MM-DDTHH:MM:SS.###Z error wcp [kubelifecycle/kubenodeconfig.go:166] [opID=########-########-####-####-####-############-vm-<ID>] Failed to get kubenode VM VirtualMachine:vm-##### configuration status. Err ServerFaultCode: A general system error occurred: vix error codes = (1, 0).

YYYY-MM-DDTHH:MM:SS.###Z error wcp [kubelifecycle/master_node_bootstrap.go:179] [opID=########-########-####-####-####-############-vm-<ID>] Failed to get guest configuration status. err API call to VMware vCenter Server (vpxd) failed. Details 'ServerFaultCode: A general system error occurred: vix error codes = (1, 0).
'
  Note: The vm-<ID> part corresponds to the vmid of the affected Supervisor Control Plane VM.

Environment

vSphere Kubernetes Service

vSphere Supervisor

Resolution

To resolve the "vix error codes = (1, 0)" error, restart the vmtoolsd service on the affected Supervisor Control Plane VM.

1. Log in to the vCenter Server Appliance (VCSA) via SSH as the root user.
2. Run the following script to retrieve the root password for the Supervisor Control Plane VMs:
   
   /usr/lib/vmware-wcp/decryptK8Pwd.py
   
   
3. Log in to the Host Client for the ESXi host where the affected Supervisor Control Plane VM is residing.
4. In the left navigator, select Virtual Machines.
5. Locate the affected Supervisor Control Plane VM and open the VM Console.
6. Log in to the guest OS as root using the password retrieved in Step 2.
7. Run the following command to restart the vmtoolsd service:
   
   systemctl restart vmtoolsd
   
   
8. Verify that the output of the "vix error codes = (1, 0)." error in /var/log/vmware/wcp/wcpsvc.log has stopped and that the upgrade has resumed from the vSphere Client.

Note: Even after resolving the "vix error codes = (1, 0)." error, the upgrade may remain stalled due to another issue described in KB 382941.
In that case, please refer to the article above.

Additional Information

• Supervisor Cluster Deployment Stalls During "Configured Core Supervisor Services" (382941)
• Troubleshooting vSphere Supervisor Control Plane VMs (323407)
• スーパーバイザー クラスタのアップグレードが「ServerFaultCode: A general system error occurred: vix error codes = (1, 0).」により完了しない (454214)