"Failed to load OpenSSL libraries" rhttpproxy service crashes continuously during in-place windows vcenter server upgrade from 6.0 U3i to 6.0 U3j

Products

• VMware vCenter Server

Issue/Introduction

This article is to identify and resolve the rhttpproxy service crashing issue while upgrading to 6.0U3j.

Symptoms:

• While updating the Windows vCenter from 6.0U3i to 6.0U3j the rhttpproxy service fails to start, however it tries to start continuously in an infinite loop, every time it fails to start it creates a dump file as the service was crashing.
• You will see similar entries in rhttpproxy.log as mentioned below,

rhttpproxy.log:

2019-11-01T11:13:14.563+01:00 Section for VMware VirtualCenter, pid=7044, version=6.0.0, build=14510545, option=Release
2019-11-01T11:13:14.563+01:00 verbose rhttpproxy[08932] [Originator@6876 sub=Default] Dumping early logs:

------ Early init logs start --------
2019-11-01T11:13:14.345+01:00 info -[08932] [Originator@6876 sub=Default] Initialized SystemFactory

------ Early init logs end  --------
2019-11-01T11:13:14.563+01:00 warning rhttpproxy[08932] [Originator@6876 sub=Default] Unrecognized log/level '' using 'info'
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Logging uses fast path: true
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Initialized channel manager
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Current working directory: C:\Windows\system32
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] ThreadPool windowsStackImmediateCommit = true
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08328] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-01T11:13:14.563+01:00 info rhttpproxy[12756] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-01T11:13:14.563+01:00 info rhttpproxy[09208] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=ThreadPool] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 44, 2, 18, 2147483647
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-01T11:13:14.563+01:00 info rhttpproxy[11588] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Listening on HTTP port 80.
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Listening on HTTPS port 443.
2019-11-01T11:13:14.563+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] Vmacore::InitSSL: path configuredLibDir = C:\\Program Files\\VMware\\vCenter Server\openSSL\
2019-11-01T11:13:14.797+01:00 info rhttpproxy[08932] [Originator@6876 sub=Default] CoreDump: Writing minidump
2019-11-01T11:13:14.953+01:00 panic rhttpproxy[08932] [Originator@6876 sub=Default] 

--> Panic: SSLLoadSharedLibraries: Failed to load OpenSSL libraries. libdir is C:\\Program Files\\VMware\\vCenter Server\openSSL\
--> 
--> Backtrace:
--> 
--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-14510545, tag: rhttpproxy
--> backtrace[00] vmacore.dll[0x001C9C0A]
--> backtrace[01] vmacore.dll[0x0005E15F]
--> backtrace[02] vmacore.dll[0x0005EA7E]
--> backtrace[03] vmacore.dll[0x001DF985]
--> backtrace[04] vmacore.dll[0x001DFA7D]
--> backtrace[05] vmacore.dll[0x00027B53]
--> backtrace[06] vmacore.dll[0x001FCE38]
--> backtrace[07] vmacore.dll[0x00206E9E]
--> backtrace[08] vmacore.dll[0x00118A7F]
--> backtrace[09] rhttpproxy.exe[0x0000401A]
--> backtrace[10] rhttpproxy.exe[0x000017B3]
--> backtrace[11] sechost.dll[0x0000A82D]
--> backtrace[12] kernel32.dll[0x0001556D]
--> backtrace[13] ntdll.dll[0x0005385D]
--> [backtrace end]
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.0.x

Cause

Rhttpproxy failed to load openssl libraries while starting rhttpproxy services as the OpenSSL libraries were used by third party applications.

Resolution

• To resolve this issue make sure there are no third party applications which uses the same openssl libraries as vCenter Server and uninstall the applications.
• Re-try the windows in-place vCenter Upgrade.