Connecting ESXi host back to vCenter after reverting to the snapshot fails with "License not available to perform the operation License file download from Hostxxxx to vcenter failed due to Exception Fault cause: vmodl.RuntimeFault"

Products

• VMware vCenter Server

Issue/Introduction

- Hosts were in disconnected state in vCenter. Trying to connect the hosts failed with the following message :

- We see below in the vpxd.log :

####-##-##T##:##:##.###+##:## info vpxd[40449] [Originator@6876 sub=VCDlfDownloader opID=LicenseClientInit-fc4111] Download vc shipped dlfs from '/usr/lib/vmware-vpx//licenses/site/'. VC Indicator Directory '/etc/vmware-vpx//licenses/site//VMware VirtualCenter Server/7.0/7.0.5.0'
####-##-##T##:##:##.###+##:## info vpxd[40449] [Originator@6876 sub=vmomi.soapStub[8] opID=LicenseClientInit-fc4111] SOAP request returned HTTP failure; <SSL(<io_obj p:0x0000560b697cc198, h:82, <TCP 'xxx.x.x.x : 41726'>, <TCP 'xxx.x.x.x : 443'>>), /ls/sdk>, method: uploadLicenseFile; code: 500(Internal Server Error); fault: (vmodl.RuntimeFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x0000560b697cc198, h:82, <TCP 'xxx.x.x.x: 41726'>, <TCP 'xxx.x.x.x : 443'>>), /ls/sdk>]: uploadLicenseFile
--> Unable to store license files"
--> }
####-##-##T##:##:##.###+##:## error vpxd[40449] [Originator@6876 sub=licenseClient opID=LicenseClientInit-fc4111] Downloading of VC install DLFs failed with N5Vmomi12RuntimeFault9ExceptionE(Fault cause: vmodl.RuntimeFault
--> )
--> [context]zKq7AVECAQAAAOIfcwEddnB4ZAAAAto3bGlidm1hY29yZS5zbwAAmXksABdtLQAf6jIBvRoObGlidm1vbWkuc28AAeNtDAEqGxUBPCUVAY66FIIerS8BdnB4ZAABuboQAbi1EAPjcP1saWJ2aW0tdHlwZXMuc28AgheuLQGCVpwuAYKdPC4BgoZeLgGCsMwuAYLUzS4Bgn8PLwGCNoIuAYKmEWYBgkASZgEAkLMjAOdJIwB1nyMAwGU3BId/AGxpYnB0aHJlYWQuc28uMAAFvzYPbGliYy5zby42AA==[/context]
####-##-##T##:##:##.###+##:## error vpxd[40449] [Originator@6876 sub=licenseClientStarter opID=LicenseClientInit-fc4111] Starting license client failed with N5Vmomi12RuntimeFault9ExceptionE(Fault cause: vmodl.RuntimeFault

- VCs were in linked mode. Replication status of the affected VC was in RED in VC.

Environment

vCenter Server 7.x

Cause

Upgrade of VC failed and customer had to revert the VC to a snapshot in a ELM environment. This action is not recommended and can lead to PSC replication inconsistencies.

Resolution

NOTE : Please take a offline snapshot of both the VCs in ELM before proceeding with the below steps.

 

1. Verify the current VMdir state: 

 

   on the affected VC, Run -> /usr/lib/vmware-vmdir/bin/vdcadmintool

   Select option 6 to check the VMdir state.

   

   Ex : VMdir will show as 'Read-Only'

   

2) Check the replication status of the PSCs: 

 

   /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

   

Ex :

Host available:   Yes

Status available: No   -----------> Status says 'No'

 

My last change number:             102676

Partner has seen my change number: 99887

Partner is 2789 changes behind.  -------------------> Pending partner changes

   

2. Attempt to set the VMdir state to NORMAL: 

 

Run -> /usr/lib/vmware-vmdir/bin/vdcadmintool

 

 Select option 5 to set it to NORMAL

 

NOTE : Proceed to step 3 only if the VMdir status changes back to 'Read-Only' again.

 

3) Run the 'fixpsc.py' rebuild command to fix the inconsistencies between the PSCs as mentioned in KB316566

 

4) Verify the VMdir state again: 

 

   /usr/lib/vmware-vmdir/bin/vdcadmintool

 

   Select option 6 to check the VMdir state.

   

5) Check the replication status: 

 

   /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

   

   Replication status of the PSCs should say :

   Host available:   Yes

   Status available: Yes       ------> Status should say 'Yes'

   Partner is 0 changes behind   ------> Partner 0 changes behind

   

6) Login to vCenter. Start connecting the hosts back to the vCenter and they should connect successfully.