Access to VC fails with "no healthy upstream".

Products

• VMware vCenter Server

Issue/Introduction

This article will help users to recover from this crash and get access back to VC.

Symptoms:

Access to VC fails with "no healthy upstream". Several VC services, including wcp crashes. wcp service, leave a core file. 

 

Environment

VMware vCenter Server 8.0.2

Cause

After a VC trusted roots change and when wcpsvc tries to re-authenticate with VC it goes into an infinite loop and eventually crashes. Due to the infinite recursion of wcp some other VC services run out of memory and crash.

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:

Follow the below workaround:

• To Remove the core file:

 

1. SSH to vcenter as a root user and get the bash shell
 
Command> shell
 
root@vcenter [ ~ ]#
 
2. Check the storage saturation
 
root@vcenter [ ~ ]# df -h
 
/dev/mapper/core_vg-core                   49G   49G     0 100% /storage/core <<<<<<< 100% full
 
3. Locate and Remove the core file : core.wcpsvc.<PID>
 
root@vcenter [ ~ ]#  rm -rf /var/core/core.wcpsvc.31091 

 

• Check the status of unhealthy wcp service and its dependent services recursively

 For example:
 
root@vcenter [ ~ ]# vmon-cli -s wcp
 
Name: wcp
 
Starttype: AUTOMATIC
 
RunState: STOPPED
 
RunAsUser: wcp
 
CurrentRunStateDuration(ms): 30407747
 
HealthState: UNHEALTHY <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 
FailStop: TRUE
 
MainProcessId: N/A
 
root@vcenter [ ~ ]# vmon-cli --list-dependencies wcp
 
cis-license
 
eam
 
lookupsvc
 
sts <<<<<<<<<<<<<<<<<<<<<< This will be UNHEALTHY
 
trustmanagement
 
vapi-endpoint <<<<<<<<<<<< This will be in INITIALIZED State
 
vmware-vpostgres
 
vpxd
 
vpxd-svcs
 
 
root@vcenter [ ~ ]# vmon-cli --list-dependencies sts
   
Identify dependent services that are unhealthy
 

• Restart all the services that are unhealthy: e.g. sts, vapi-endpoint, wcp

 

root@vcenter [ ~ ]# vmon-cli -r sts
 
root@vcenter [ ~ ]# vmon-cli -r vapi-endpoint
 
root@vcenter [ ~ ]# vmon-cli -r wcp
 

• Check Status of the restarted services

root@vcenter [ ~ ]# vmon-cli -s sts
 
Name: sts
 
Starttype: AUTOMATIC
 
RunState: STARTED
 
RunAsUser: sts
 
CurrentRunStateDuration(ms): 98627165
 
HealthState: HEALTHY
 
FailStop: N/A
 
MainProcessId: 108114
 
root@vcenter [ ~ ]# vmon-cli -s vapi-endpoint
 
Name: vapi-endpoint
 
Starttype: AUTOMATIC
 
RunState: STARTED
 
RunAsUser: vapiEndpoint
 
CurrentRunStateDuration(ms): 98735351
 
HealthState: HEALTHY
 
FailStop: N/A
 
MainProcessId: 107948
 
root@vcenter [ ~ ]# vmon-cli -s wcp
 
Name: wcp
 
Starttype: AUTOMATIC
 
RunState: STARTED
 
RunAsUser: wcp
 
CurrentRunStateDuration(ms): 99301946
 
HealthState: HEALTHY
 
FailStop: N/A
 
MainProcessId: 119240