Could not connect to one or more vCenter Server Systems: https://VC_FQDN_OR_IP: 443/sdk error in the vSphere Client
Article ID: 312682
Updated On:
Products
Issue/Introduction
This article explains, how to check and fix the problem with linked mode vCenter. Confirm the linked mode data table structure from the LDAP level first, and then solve the front-end problem.
Symptoms:
There are two or more vCenter Servers in a Linked Mode;
When logged into the vSphere Client for vCenter server A, you can see vCenter server A, vCenter server B, vCenter server C, vCenter server D in the inventory, but when logged into vCenter server B,or vCenter server C, or vCenter server D, you cannot see vCenter server A, only can see vCenter server B, vCenter server C, vCenter server D;
The vSphere Client displays the error as follows:
could not connect to one or more vCenter Server Systems:https://vCenter_A:443/sdk
At vCenter server B, vCenter server C, vCenter server D, In the /var/log/vmware/vsphere-client/vsphere_client_virgo.log file, you see entries similar to the following:
[2023-03-21T12:13:32.293+08:00] [ERROR] vc-service-async-pool-74933 c.v.v.v.extension.propertycollector.VcExtensionPropertyCollector Exception while listening for extension updates. VcService = https://vcenterA.xxxxx.local:443/sdk (6d288b68-9d3a-4f13-afdb-7928d08b281e), dataVersion = 1. com.vmware.vim.vmomi.client.exception.InvalidSslCertificateException: Invalid SSL certificate (HTTP 526 status code)
[2023-03-21T12:14:28.925+08:00] [ERROR] wcp-plugins-notifier-1 c.v.v.plugin.notification.WcpServicePluginNotificationsListener Unable to subscribe to vcenterA.xxxxx.local (6d288b68-9d3a-4f13-afdb-7928d08b281e), nodeId: 9acf4acc-8890-11eb-a146-000c29a8f3f5 java.lang.IllegalStateException: javax.websocket.DeploymentException: The HTTP response from the server [526] did not permit the HTTP upgrade to WebSocket
.....
Caused by: javax.websocket.DeploymentException: The HTTP response from the server [503] did not permit the HTTP upgrade to WebSocket
Environment
Cause
4 vCenter was upgraded from 6. x to 7.0, and no unregistration was done for the old PSC after the upgrade. Although the 4 vCenters were running normally on the surface at that time, and the vCenter could keep running normally without any major changes, once operations such as certificate refresh were issued, all of them would disrupt the balance of this state, leading to failure.
Resolution
1. Use lsdoctor to check the vCenter in linked mode, and if there is a problem, fix it according to lsdoctor output. Please take a snapshot with memory for all vCenter before fixing.
For lsdoctor usage, please refer to KB: Using the 'lsdoctor' Tool .
2. Use JXplorer to check the replication agreement relationship of linked mode vCenter.
For how to connect to vCenter using JXplorer, please refer to KB: Using JXplorer to connect to the vSphere Single Sign-On
After connecting to JXplorer, go through the tree diagram and click vcenter> Configuration > Sites > default-site or Default-First-Site > Servers
Under Servers of default-site, the PSC addresses pointed to during vCenter 6. x linked mode are listed, check if this directory includes all current vCenter addresses.
Under Servers in Default-First-Site, the PSC addresses pointed to by vCenter 7.0 linked mode are listed. Check whether the values pointed to under each vCenter in this directory are the addresses of the old PSC.
If either or both of the above two checks are yes. This means that the PSC has not been unregistered after the upgrade.
SSH to each vCenter and execute the unregister command for the old PSC:
cmsso-util unregister --node-pnid Platform_Services_Controller_System_Name --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password
Please refer to KB for details: Decommission of external PSC after successful converge upgrade/migration
Note: The correct topology diagram should be empty for default-site, and LDAP addresses of other vCenter should exist under each vCenter under Default-First-Site > Servers (mesh topology), or only the LDAP address of linked mode master node exists, and the LDAP addresses of the remaining vCenter exist under the master node. LDAP addresses of the remaining vCenter under the master node. (star topology)
Mesh topology example
Default-First-Site | Servers- |-vcenterA.xxxxx.local | |-ldap://vcenterB.xxxxx.local | |-ldap://vcenterC.xxxxx.local | |-ldap://vcenterD.xxxxx.local | |-vcenterB.xxxxx.local | |-ldap://vcenterA.xxxxx.local | |-ldap://vcenterC.xxxxx.local | |-ldap://vcenterD.xxxxx.local | |-vcenterC.xxxxx.local | |-ldap://vcenterA.xxxxx.local | |-ldap://vcenterB.xxxxx.local | |-ldap://vcenterD.xxxxx.local | |-vcenterD.xxxxx.local | |-ldap://vcenterA.xxxxx.local | |-ldap://vcenterB.xxxxx.local | |-ldap://vcenterC.xxxxx.local
Star topology example, vcenterA is linked-mode master node
Default-First-Site | Servers- |-vcenterA.xxxxx.local | |-ldap://vcenterB.xxxxx.local | |-ldap://vcenterC.xxxxx.local | |-ldap://vcenterD.xxxxx.local | |-vcenterB.xxxxx.local | |-ldap://vcenterA.xxxxx.local | |-vcenterC.xxxxx.local | |-ldap://vcenterA.xxxxx.local | |-vcenterD.xxxxx.local |-ldap://vcenterA.xxxxx.local
Ring Topology
Default-First-Site
|
Servers-
|-vcenterA.xxxxx.local
| |-ldap://vcenterB.xxxxx.local
| |-ldap://vcenterD.xxxxx.local
|-vcenterB.xxxxx.local
| |-ldap://vcenterA.xxxxx.local
| |-ldap://vcenterC.xxxxx.local
|-vcenterC.xxxxx.local
| |-ldap://vcenterA.xxxxx.local
| |-ldap://vcenterD.xxxxx.local
|-vcenterD.xxxxx.local
| |-ldap://vcenterA.xxxxx.local
| |-ldap://vcenterC.xxxxx.local
3. Fix the LDAP data table structure of linked mode vCenter
Here we choose a star topology, with vcenterA as the master node of linked mode. The following command is executed on vcenterA only:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u administrator -w Administrator_Password
For recreating replication relationships, please refer to KB: Determining replication agreements and status with the Platform Services Controller(PSC)
For example:
vdcrepadmin -f createagreement -2 -h vcenterA.xxxxx.local -H vcenterB.xxxxx.local -u administrator -w Administrator_Password
vdcrepadmin -f createagreement -2 -h vcenterA.xxxxx.local -H vcenterC.xxxxx.local -u administrator -w Administrator_Password
vdcrepadmin -f createagreement -2 -h vcenterA.xxxxx.local -H vcenterD.xxxxx.local -u administrator -w Administrator_Password
Use JXplorer again to check the replication agreement relationship of linked mode vCenters, the topology diagram at this time should be the star topology example above.
4. Check the linked mode partnership status on all
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w passwd
Under star topology, the output in master node vcenterA should look similar to the following:
Partner: vcenterB.xxxxx.local
Host available: Yes
Status available: Yes
My last change number: 9502
Partner has seen my change number: 9502
Partner is 0 changes behind.
Partner: vcenterC.xxxxx.local
Host available: Yes
Status available: Yes
My last change number: 9502
Partner has seen my change number: 9502
Partner is 0 changes behind.
Partner: vcenterD.xxxxx.local
Host available: Yes
Status available: Yes
My last change number: 9502
Partner has seen my change number: 9502
Partner is 0 changes behind.
The rest of vCenter's output should be
Partner: vcenterA.xxxxx.local
Host available: Yes
Status available: Yes
My last change number: 9502
Partner has seen my change number: 9502
Partner is 0 changes behind.
(If the individual vCenter status is incorrect, check the LDAP data structure again and make sure it is correct, then please do a shutdown snapshot for all vCenter first. Ignore the status issue for now and move on to the next fix)
5. Perform a certificate refresh operation for vCenter.
Before refreshing the certificate, please take a shutdown snapshot for all vCenters or do a file-based backup at the same time.
Shut down the vCenter virtual machines except for the master node (vcenterA), refresh the STS certificate in vcenterA first, and then refresh the Machine SSL and Solution User certificates
Refer to KB: "Signing certificate is not valid" error in VCSA 6.5.x/6.7.x and vCenter Server 7.0.x and How to use vSphere Certificate Manager to Replace SSL Certificates
6. After the certificate refresh operation of vcenterA is completed and the services are running normally, then open other vCenter and let them exchange data and certificates with vcenterA.
7. When all vCenter services are in normal operation, open the vSphere Client again. At this time, the links to all vCenter services are back to normal.
Additional Information
Impact/Risks:
The vCenter in linked mode will be disconnected from each other. They can't see each other or can only see each other unilaterally.
Feedback
