Autodeployed ESXi host fails with error: Error loading /vmw/rbd/host//waiter.tgz Fatal error: 15 (Not found)

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi host deployment with Autodeploy fails during the boot process with an error:
  
  
  Loading /vmw/rbd/host/d2bg88d4f79102c7921gc29g30647ffg/waiter.tgz
  Error loading /vmw/rbd/host/ d2bg88d4f79102c7921gc29g30647ffg /waiter.tgz
  Fatal error: 15 (Not found)
  
  
• In the /var/log/vmware/rbd/rbd-cgi.log file, you see error similar to:
  
  
  2016-11-17T14:11:10.991 [4124]INFO:sslcert:cert files are missing from /var/lib/rbd/ssl/host-13

  2016-11-17T14:11:10.991 [4124]INFO:sslcert:cert files are missing from /var/lib/rbd/ssl/d2bg88d4f79102c7921gc29g30647ffg

  2016-11-17T14:11:10.993 [4124]INFO:sslcert:cert files are missing from /var/lib/rbd/ssl/host-13

  2016-11-17T14:11:10.994 [4124]INFO:sslcert:Generating SSL cert for d2bg88d4f79102c7921gc29g30647ffg (esxi01.domain.com)

  2016-11-17T14:11:11.778 [4124]INFO:sslcert:Validating certificate checks for hostId: d2bg88d4f79102c7921gc29g30647ffg

  2016-11-17T14:11:11.781 [4124]INFO:sslcert:Days left for expiry: 230 days

  2016-11-17T14:11:11.782 [4124]ERROR:plugins:exception:rbdplugins.sslcert.vmwWaiterTgz -- The days left for certificate expiry is less than thethreshold value, Days_left:230, Configured_threshold:240

  Traceback (most recent call last):

  File "/build/mts/release/bora-3437678/bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/plugins.py", line 220, in _curry

  File "/build/mts/release/bora-3437678/bora/install/vmvisor/autodeploy/var/rbdplugins/sslcert.py", line 315, in vmwWaiterTgz

  File "/build/mts/release/bora-3437678/bora/install/vmvisor/autodeploy/var/rbdplugins/sslcert.py", line 187, in _validateCert

  Exception: The days left for certificate expiry is less than thethreshold value, Days_left:230, Configured_threshold:240
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  
  
• On an external Platform Services Controller, using vecs-cli to check the machine and vpxd-extension certificates on the vCenter Server shows that they are not due to expire within the next 240 days. For more information, see Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.0 (2111411).
  
  
• Checking the certificates on the Platform Services Controller show that they are due to expire within the 240 day threshold.

Environment

VMware vSphere ESXi 6.0
VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.0.x

Cause

This issue occurs when the certificate refresh threshold is greater than the remaining expiry time on the certificates being created for the ESXi hosts.

When an ESXi host is added to a vCenter Server, it sends a certificate signing request for the host to the VMCA (provided you are using VMCA and not using custom certs for each ESXi host and pre-populating the /var/lib/rbd/ssl/host-id directory). The VMCA can only issue certificates which are valid up to the expiry date of either the machine certificate, or any intermediate certificate in the chain to your Root CA. The default value for the certificate refresh threshold is 240 days.

Resolution

To resolve this issue, use one of these options:

• Replace the VMCA certificate so that it is no longer within the default 240 day expiry time frame. For more information, see Replacing default certificates with CA signed SSL certificates in vSphere 6.x (2111219).
• Reduce the default 240 day expiry time to a value less than the current certificate expiry time frame. For more information, see ESXi Certificate Default Settings section in VMware vSphere 6.0 Documentation Center.

Additional Information

Replacing default certificates with CA signed SSL certificates in vSphere 6.x
Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.0