Restarting the HMS service within VMware Cloud Foundation causes any Workflow containing TOR configurations to fail

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Unable to expand workload domains
• ​Unable to add new hosts to inventory
• Unable to add a new datacenter network to Workload Domains.
• You see entries in the /opt/vmware/bringup/logs/evosddc-bringup-debug.log file on the SDDC Manager Controller VM similar to the following:

2018-08-28 14:40:27.058 DEBUG [http-nio-8080-exec-9] [com.vmware.vrack.hms.rest.factory.URIBuilder] creating URI for the path /api/1.0/hms/switches/R1S2/vlans, query null
2018-08-28 14:40:27.059 DEBUG [http-nio-8080-exec-9] [com.vmware.vrack.hms.rest.factory.HmsOobAgentRestTemplate] Created PUT request for "http://localhost:8448/api/1.0/hms/switches/R1S2/vlans"
2018-08-28 14:40:27.066 DEBUG [http-nio-8080-exec-9] [com.vmware.vrack.hms.rest.factory.HmsOobAgentRestTemplate] PUT request for "http://localhost:8448/api/1.0/hms/switches/R1S2/vlans" resulted in 404 (Not Found); invoking error handler
2018-08-28 14:40:27.067 INFO [http-nio-8080-exec-9] [com.vmware.vrack.hms.controller.MaintenanceInterceptor] Returning response code 404 for request: [ PUT on /hms-local/api/1.0/hms/switches/R1S2/vlans ]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 2.3.x

Cause

Restarting the hms service without restarting the vcfmanager service causes an inconsistency in the hms inventory and results in API failures to the hms process and possibly other unobserved issues.

Resolution

• To fix a current issue, restart vcfmanager service by using the follow command on the SDDC manager controller VM

systemctl restart vcfmanager

• In the future, do not restart the hms service without restarting vcfmanager afterwards.