VMware Cloud Director cell service startup stuck at 54%

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director(i.e. VCD) cell service startup stuck at 54% when com.vmware.vcloud.backend-core service fails to start
• /opt/vmware/vcloud-director/logs/cell.log reports:
  
  Application initialization detailed status report: 54% complete
    com.vmware.vcloud.vcell-runtime-impl                         Subsystem Status: [COMPLETE]
    com.vmware.vcloud-common-cell-impl                           Subsystem Status: [COMPLETE]
    com.vmware.vcloud.common-util                                Subsystem Status: [COMPLETE]
    com.vmware.vcloud.api-framework                              Subsystem Status: [COMPLETE]
    com.vmware.vcloud.common-vmomi                               Subsystem Status: [COMPLETE]
    com.vmware.pbm.placementengine                               Subsystem Status: [COMPLETE]
    com.vmware.vcloud.vim-proxy                                  Subsystem Status: [COMPLETE]
    com.vmware.vcloud.fabric.foundation                          Subsystem Status: [COMPLETE]
    com.vmware.vcloud.imagetransfer-server                       Subsystem Status: [COMPLETE]
    com.vmware.vcloud.fabric.storage                             Subsystem Status: [COMPLETE]
    com.vmware.vcloud.fabric.compute                             Subsystem Status: [COMPLETE]
    com.vmware.vcloud.fabric.net                                 Subsystem Status: [COMPLETE]
    com.vmware.vcloud.backend-core                               Subsystem Status: [WAITING]
    com.vmware.vcloud.faas-behaviors-aws-lambda                  Subsystem Status: [WAITING]
    com.vmware.vcloud.vapp-lifecycle                             Subsystem Status: [WAITING]
    com.vmware.vcloud.content-library                            Subsystem Status: [WAITING]
    com.vmware.vcloud.presentation-api-impl                      Subsystem Status: [WAITING]
    com.vmware.vcloud.statsfeeder-application                    Subsystem Status: [WAITING]
    com.vmware.vcloud.metrics-core                               Subsystem Status: [WAITING]
    com.vmware.vcloud.multi-site-core                            Subsystem Status: [WAITING]
    com.vmware.vcloud.multi-site-api                             Subsystem Status: [WAITING]
    com.vmware.vcloud.rest-api-handlers                          Subsystem Status: [WAITING]
  
  
• /opt/vmware/vcloud-director/logs/vmware-vcd-watchdog.log shows watchdog attempts to restart the cell service approximately every 5 minutes:
  
  YYYY-MM-DD 16:59:53 | ALERT | vmware-vcd-cell is dead but /var/run/vmware-vcd-cell.pid exists, attempting to restart it
YYYY-MM-DD 17:00:03 | INFO  | Started vmware-vcd-cell (pid=#####)
YYYY-MM-DD 17:00:03 | WARN  | wget failed, return code: 4, will retry in 30 seconds
YYYY-MM-DD 17:00:34 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:01:34 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:02:34 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:03:34 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:04:35 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:05:35 | ALERT | vmware-vcd-cell is dead but /var/run/vmware-vcd-cell.pid exists, attempting to restart it
YYYY-MM-DD 17:05:45 | INFO  | Started vmware-vcd-cell (pid=#####)
YYYY-MM-DD 17:05:45 | WARN  | wget failed, return code: 4, will retry in 30 seconds
YYYY-MM-DD 17:06:16 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:07:16 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:08:16 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:09:16 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:10:17 | WARN  | Server status returned HTTP/1.1 503
YYYY-MM-DD 17:11:17 | ALERT | vmware-vcd-cell is dead but /var/run/vmware-vcd-cell.pid exists, attempting to restart it
YYYY-MM-DD 17:11:27 | INFO  | Started vmware-vcd-cell (pid=#####)


• /opt/vmware/vcloud-director/logs/cell-runtime.log has below messages:
  
  Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vcdSecuredWebappConfiguration': Unsatisfied dependency expressed through field 'sessionFilter'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionFilter': Unsatisfied dependency expressed through field 'filterUtils'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'filterUtils': Unsatisfied dependency expressed through field 'securityService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityService': FactoryBean threw exception on object creation; nested exception is java.util.concurrent.TimeoutException: Timed out waiting for service: 'securityService', objectClasses='[interface com.vmware.vcloud.api.presentation.service.SecurityService]', filter='(objectClass=com.vmware.vcloud.api.presentation.service.SecurityService)' 

Environment

VMware Cloud Director 10.5.1.1
VMware Cloud Director 10.6.1
VMware Cloud Director 10.6.1.1

Cause

VCD cell service startup has a default hard-coded timeout value of 5 minutes. If the service does not start within the time limit the watchdog service will restart cell services automatically.

Resolution

The timeout value will be increased from 5 minutes to 10 minutes in future releases. In current release there's no way to modify the timeout value. 

Workaround would be to move the VCD from current datastore/cluster to another datastore/cluster with more resources so that the VCD cell services can boot up in a quicker manner.

Additional Information

The service startup failure can also be a result of undersized cell, refer to: "An error occurred during the initialization. Accessing the application through an unsupported public URL or poor connectivity might cause this error" when accessing provider portal for Cloud Director