Transport node upgrade from VMware NSX 4.0.1 fails when service account has expired - Error code:400

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are upgrading from NSX 4.0.1 to NSX 4.0.2.
• The cluster is managed using vSphere Lifecycle Manager.
• The upgrade coordinator has been upgraded and the transports nodes are staged for installation.
• The host upgrade fails with "General error (Error code:400)".
• Checking the log support bundle under /var/log/upgrade-coordinator/upgrade-coordinator.log you see the below error:

ERROR http-nio-127.0.0.1-7442-exec-5 InventoryRestClientImpl 14418 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP7044" level="ERROR" subcomp="upgrade-coordinator"] Error in rest call url= /cm-inventory/api/v1/fabric/compute-managers/<uuid>?action=change-service-account-password , method= POST , response= {
    "module_name" : "common-services",
    "error_message" : "The requested with given URI, HTTP method and set of parameters cannot be processed.",
    "error_code" : 269
}
 , error= [{"errorMessage":"The requested with given URI, HTTP method and set of parameters cannot be processed.","errorData":{"moduleName":"common-services","errorCode":"269","statusCode":"NOT_FOUND"}}]
org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{<EOL> "module_name" : "common-services",<EOL> "error_message" : "The requested with given URI, HTTP method and set of parameters cannot be processed.",<EOL> "error_code" : 269<EOL>}<EOL>"

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.0.0.1

Cause

• During the upgrade process new VIBs are installed on the host transport node. This requires vCenter authentication.
• If the service account password expiry is detected the "change-service-account-password" API is called by the manager.
• This API call fails with error "NotFound: 404" as this call in not available in NSX 4.0.1.

Resolution

This is resolved in NSX version 4.0.2 VMware Downloads.

Workaround:

• To workaround this issue resolve the service account password expiration:
  • This can be accomplished by going to 'Compute manager' under 'System' in NSX UI.
  • Click on Edit next to 'FQDN or IP address' of the Compute manager.
  • Re-enter the vSphere username and password used to register it with NSX-T.
  • This will re-create the service account, and trigger a Full inventory sync from vCenter.
• If the above steps do not resolve the error, please the restart proton service on all the NSX Managers. Restarting proton on all the managers will clear out the cache and NSX Manager will be forced to make a new API Call for vAPI Token. 
  • Check the cluster status is healthy 'get cluster status' and proceed to next step, if status is healthy.
  • As admin user in the CLI of the first manager, run the command 'restart service manager'.
  • Check the cluster status is healthy again 'get cluster status' once restart has completed
  • Repeat on the next manager.
• If the error persists please restart the upgrade coordinator service:
  • From admin of the CLI on each manager run the commands "restart service install-upgrade".