VCF Automation: vmware-system-user account showing 'Disconnected' Status in VCF Operations UI

Products

• VMware vSphere ESXi

Issue/Introduction

• The vmware-system-user account appears as "Disconnected" in the VCF Operations user interface.
• Attempts to remediate the password for VCF Automation or Fleet Management appliances fail with the error message: error=403 : [no body] and/or error code LCMVMSP10035

Environment

• VMware Cloud Foundation 9.0.x
• VCF Components: VCF Automation 9.0.x, Fleet Management Appliance 9.0.x

Cause

• Disk Exhaustion: The /sdaX partition on the VCF Automation/Fleet Management appliance becomes 100% full due to the accumulation of API log files (e.g., api.log).
• This exhaustion prevents background login services and authentication daemons from writing temporary files, causing the services to hang or crash.

Resolution

• Take a snapshot of the VCF Automation and Fleet Management appliance VMs.
• Log in to the VCF Automation appliance CLI and identify the partition causing the saturation: df -h
• Navigate to the root directory and filter for the largest file(s): du -hxd1 | sort -h
• Navigate to the directory identified as having the largest log file consumption and truncate the excessive logs: > <file_name>
• Retry the remediation from the VCF Operations UI.
• If remediation continues to fail despite clearing space  perform a manual password reset via the VCFA CLI on eaach node: e.g. passwd vmware-system-user
• Try the remediation again.

Additional Information

Error: "Password Expired" During SSH Authentication to VCF Automation or VCF Identity Broker 

Generate a Support Bundle for VCF Fleet Management