Component Manager services fail at starting Content Library Service

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server Appliance 6.0 is running at Update 2.
• The vSphere Web Client reports 503 error.
• Running the "service-control --start --all" fails to complete:
  
  Stdout = Starting VMware Content Library Service...
wrapper | An encoding declaration is missing from the top of configuration file, /etc/vmware/wrapper_common.conf, trying the system encoding.
wrapper | An encoding declaration is missing from the top of configuration file, /etc/vmware/wrapper_linux.conf, trying the system encoding.
wrapper | Spawning intermediate process...
Waiting for VMware Content Library Service..................
WARNING: VMware Content Library Service may have failed to start.
Last login:

Stderr =
{
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-vdcs', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vdcs', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
 
• In the /var/log/vmware/cm/wrapper.log file, the following log entries appear:
  
  INFO | jvm 1 | INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: ../lib:../../bin
INFO | jvm 1 | org.apache.coyote.AbstractProtocol init
INFO | jvm 1 | INFO: Initializing ProtocolHandler ["http-bio-127.0.0.1-18090"]
INFO | jvm 1 | org.apache.coyote.AbstractProtocol init
INFO | jvm 1 | SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-127.0.0.1-18090"]
INFO | jvm 1 | java.net.BindException: Address already in use /127.0.0.1:18090
INFO | jvm 1 | at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:411)
INFO | jvm 1 | at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:646)
INFO | jvm 1 | at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
INFO | jvm 1 | at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119
   
• In the /var/log/vmware/vdcs/wrapper.log file, the following log entries appear:
  
  DEBUG | wrapper | An encoding declaration is missing from the top of configuration file, /etc/vmware/wrapper_linux.conf, trying the system encoding.
  DEBUG | wrapper | active log file changed: /var/log/vmware/vdcs/wrapper.log
  DEBUG | wrapper | active log file changed: /var/log/vmware/vdcs/wrapper.log
  ERROR | wrapper | 13104 pid file, /var/log/vmware/vdcs/vmware-vdcs.pid, already exists.
  FATAL | wrapper | ERROR: Could not write pid file /var/log/vmware/vdcs/vmware-vdcs.pid: Inappropriate ioctl for device wrapper.log lines 36-66/66 (END)

Environment

VMware vCenter Server 6.0.x

Cause

This issue is occurring as there is a corrupted or stale PID for the content library service which is has not been cleared from the previous running state.

Resolution

To resolve this issue, remove the vmware-vdcs.pid from the vCenter Server Appliance: 

1. Connect to the vCenter Server Appliance using SSH and root credentials.
2. Type shell and press Enter.
3. Navigate to /var/log/vmware/vdcs using this command:
   
   cd /var/log/vmware/vdcs


4. Run the ls command and verify it there is a vmware-vdcs.pid entry
5. Run this command to create a new folder to move the pid file to:
   
   mkdir clspid
   
   
6. Move the vmware-vdcs.pid file using this command:
   
   mv vmware-vdcs.pid clspid 
   
   
7. Run this command to start the vCenter Server Appliance services
   
   service-control --start --all