8 precheck errors for Upgrading vCenter to 9.X/ 9.1 Root password/ vCenter backup/ vCenter storage health/ vCenter cloud system health/ vCenter NTP/ vCenter to SDDC Manager time difference/ Lookup service availability/ Vpxd service availability
search cancel

8 precheck errors for Upgrading vCenter to 9.X/ 9.1 Root password/ vCenter backup/ vCenter storage health/ vCenter cloud system health/ vCenter NTP/ vCenter to SDDC Manager time difference/ Lookup service availability/ Vpxd service availability

book

Article ID: 448583

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Running precheck for upgrading a vCenter in a VCF 9.x/ 9.1 environment triggers the following 8 precheck errors 
Root password/ vCenter backup/ vCenter storage health/ vCenter cloud system health/ vCenter NTP/ vCenter to SDDC Manager time difference/ Lookup service availability/ Vpxd service availability

Passwords are not disconnected, NTP and VPXD services are running on the Impacted vCenter 

Operationsmanager logs show an authentication error

grep -i -C4 "Exception occurred during VC vAPI" operationsmanager/operationsmanager.log

--
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
<date && time> INFO [vcf_om,0000000000000000,0000] [c.v.v.vapi.vsphere.VcApplianceClient,pool-5-thread-6] Get the current status of the appliance update
<date && time> ERROR [vcf_om,0000000000000000,0000] [c.v.v.vapi.vsphere.VcenterVapiHelper,pool-5-thread-6] Exception occurred during VC vAPI invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = vapi.security.authorization.invalid,
defaultMessage = Unable to authorize user,

 

API calls fails for either SSO and the Service account created by SDDC manager, or only for the Service account test below

A. Run API calls from SDDC towards the impacted vCenter 
Each API call will result in a Sesison ID [alphanumeric string] 

# curl -ks https://<Impacted Vcenter>/api/session -u '[email protected]' -X POST | jq -r .
> password will be asked 
11111AAAAAAAA11111AAAAAAA

# curl -ks https://<Impacted Vcenter>/api/session -u '<svc_account_craeted _by_SDDC>@vsphere.local' -X POST | jq -r .
> password will be asked 
22222BBBBBBBBB2222BBBBBB

B. Create variables
for SSO admin
# session=11111AAAAAAAA11111AAAAAAA    
the Service account
# session2=22222BBBBBBBBB2222BBBBBB    

C. run API call from SDDC
# curl -ks https://<Impacted_Vcenter>/api/appliance/local-accounts/root -H "vmware-api-session-id: $session" | jq
{
  "warn_days_before_password_expiration": 7,
  "min_days_between_password_change": 0,
  "roles": [
    "superAdmin"
  ],
  "fullname": "root",
  "last_password_change": "<Date && Time>",
  "enabled": true,
  "has_password": true,
  "max_days_between_password_change": -1
}

# curl -ks https://<Impacted_Vcenter>/api/appliance/local-accounts/root -H "vmware-api-session-id: $session2" | jq
{
  "error_type": "UNAUTHORIZED",
  "messages": [
    {
      "args": [],
      "default_message": "Unable to authorize user",
      "id": "vapi.security.authorization.invalid"
    }
  ]
}

Depending on scenario both or the service account API call will fail 

 

Environment

VCF 9.X/ 9.1

Cause

2 scenarios observed
- multiple STS certificates present on the impacted vCenter 
- ELM broken in SDDC 5.X before upgrading, which is not supported as per documentation 9.Xand 9.1  

Resolution

A. For the 1st scenario reset the STS certificate via vCert
B. for the 2nd scenario consider:
- rollback if it's an option, i.e rollback to the point where the ELM was not broken
or
- open a VMware support case, and reference KB 448583 

Additional Information

ELM is deprecated in 9.X 9.1,. that doe not  mean unsupported.
It remains as an after upgrade action:

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/fleet-management/managing-identity-and-access-using-vcf-single-sign-on/what-is/points-to-consider-while-setting-up-vmware-cloud-foundation-sso/deactivate-enhanced-link-mode--elm--for-upgraded-vmware-cloud-foundation-vcenters.html   

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/what-is/points-to-consider-while-setting-up-vmware-cloud-foundation-sso/deactivate-enhanced-link-mode--elm--for-upgraded-vmware-cloud-foundation-vcenters.html