vSphere Web Client service is not starting after upgrading vCenter Server

Products

• VMware
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Connecting to the vSphere Web Client (https://vcenter_fqdn.com/vsphere-client) fails with a error:
  
  503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f009c095810] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)
   
• Starting the service manually fails with error:
  
  Unable to start the service
   
• In the wrapper.log file, you see entries similar to:

  STATUS | wrapperm | 2017/08/21 09:34:25 | The VMware vSphere Web Client Service is installed.

  STATUS | wrapperm | 2017/08/21 09:34:25 | Start Type: Automatic

  STATUS | wrapperm | 2017/08/21 09:34:25 | Interactive: No

  STATUS | wrapperm | 2017/08/21 09:34:25 | Running: Yes

  STATUS | wrapper | 2017/08/21 09:34:23 | --> Wrapper Started as Service

  STATUS | wrapper | 2017/08/21 09:34:23 | Java Service Wrapper Professional Edition 64-bit 3.5.24

  STATUS | wrapper | 2017/08/21 09:34:23 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

  STATUS | wrapper | 2017/08/21 09:34:23 | http://wrapper.tanukisoftware.com

  STATUS | wrapper | 2017/08/21 09:34:23 | Licensed to VMware Global, Inc. for VMware vSphere Web Client (custom)

  STATUS | wrapper | 2017/08/21 09:34:23 |

  INFO | wrapper | 2017/08/21 09:34:25 | Event Command 'wrapper_start': Command line: "C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat" -J vsphere-client -O "C:\ProgramData\VMware\vCenterServer\cfg/java/vsphere-client_jvm.conf"

  INFO | wrapper | 2017/08/21 09:34:25 | Event Command 'wrapper_start': Command launched (pid: 26012), blocking for up to 3600 seconds...

  INFO | wrapper | 2017/08/21 09:34:25 | Event Command 'wrapper_start': Command completed with exit code: 0

  INFO | wrapper | 2017/08/21 09:34:25 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.

  STATUS | wrapper | 2017/08/21 09:34:25 | Launching a JVM...

  INFO | jvm 1 | 2017/08/21 09:34:25 | WrapperManager: Initializing...

  INFO | jvm 1 | 2017/08/21 09:34:27 | VKS keystore loaded

  INFO | jvm 1 | 2017/08/21 09:34:27 | Deleted the old JKS file.

  INFO | jvm 1 | 2017/08/21 09:34:27 | JKS keystore populated successfully.

  INFO | jvm 1 | 2017/08/21 09:34:27 | JKS keystore written to file system successfully.

  INFO | jvm 1 | 2017/08/21 09:34:27 | Tomcat config file populated successfully.

  INFO | jvm 1 | 2017/08/21 09:34:27 | An error has occurred. See the log file

  INFO | jvm 1 | 2017/08/21 09:34:27 | C:\PROGRA~1\VMware\VCENTE~1\WEBCLI~1\server\work\1503272067404.log

  STATUS | wrapper | 2017/08/21 09:34:29 | <-- Wrapper Stopped

• In the Wrapper.log file, you see entries similar to:

  !SESSION Mon Aug 21 09:35:03 AEST 2017 -----------------------------------------

  !ENTRY org.eclipse.equinox.launcher 4 0 2017-08-21 09:35:03.614

  !MESSAGE Exception launching the Eclipse Platform:

  !STACK

  java.lang.ClassNotFoundException: org.eclipse.core.runtime.adaptor.EclipseStarter

  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

  at java.security.AccessController.doPrivileged(Native Method)

  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

  at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:626)

  at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)

  at org.eclipse.equinox.launcher.Main.run(Main.java:1438)

  at org.eclipse.equinox.launcher.Main.main(Main.java:1414)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

  at com.vmware.vise.vim.commons.tanuki.TanukiWrapperEx.run(TanukiWrapperEx.java:60)

  at java.lang.Thread.run(Thread.java:745)

• In the webclient.log file, you the error:
  
  [ERROR] iLogServiceListener@1c11651e org.osgi.service.log.LogService Bundle org.eclipse.equinox.ds_1.4.0.v20120112-1400, [SCR] Unexpected exception occurred! java.lang.Ill
  egalStateException: BundleContext is no longer valid
  at org.eclipse.osgi.framework.internal.core.BundleContextImpl.checkValid(BundleContextImpl.java:931)

Environment

VMware Update Manager 6.0
VMware vSphere ESXi 6.0
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.0.x
VMware vCenter Converter Standalone 6.x

Cause

This issue occurs when vSphere  Web Client is not able to initialize Eclipse due to corrupt or missing JARs or if config.ini for Eclipse if holding incorrect path about the JARs.

• config.ini is located at:
  • In appliance: /usr/lib/vmware-vsphere-client/server/configuration/config.ini
  • On Windows: C:\Program Files\VMware\vCenter Server\WebClient\server\configuration\config.ini
• JARs is located at:
  • On Appliance: /usr/lib/vmware-virgo/server/plugins/
  • On Windows: C:/PROGRA~1/VMware/VCENTE~1/virgo/server/plugins

Resolution

To resolve this issue:

1. Open the config.ini file and verify the filename and path mentioned for line starting with "osgi.framework".
2. If you find that filename or location, where org.eclipse.osgi_3.8.2.v20130124-134944.jar is located is incorrect, correct the information in config.ini file.
3. If locations are correct, it is high possibility that JARs are either missing or corrupt.
4. Copy the plugin folder from same build working vCenter Server and replace plugin folder on non-working system.
5. Start the webclient service.