vCenter Server Upgrade Fails: vc-ws1a-broker Service Fails to Start Due to Proxy Configuration Error

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

After attempting an upgrade of VMware vCenter Server from 8.0.x to 8.0 Update 3k (8.0.3.01000), the upgrade process stalls.
The vc-ws1a-broker service fails to initialize, preventing the environment from completing the upgrade.
The system reports the following error message during the upgrade or subsequent service start attempts: Failed to start vc-ws1a-broker services. Error: A system error occurred.
Log analysis in vmon.log reveals a container runtime initialization failure: error mounting "/etc/sysconfig/proxy" to rootfs at "/etc/sysconfig/proxy": ... openat2 /etc/sysconfig/proxy: not a directory

YYYY-MM-DDTHH:MM:SS.942Z Wa(03) host-XXXXXXX <vc-ws1a-broker> Service pre-start command's stderr: YYYY-MM-DD HH:MM:SS MainThread INFO Executing command ['/usr/bin/runc', 'run', '-b', '/storage/containers/ws1-init-container/XXXXXXX', 'ws1-init-container']
YYYY-MM-DDTHH:MM:SS.942Z Wa(03)+ host-XXXXXXX
YYYY-MM-DDTHH:MM:SS.979Z Wa(03) host-XXXXXXX <vc-ws1a-broker> Service pre-start command's stderr: YYYY-MM-DD HH:MM:SS MainThread INFO Command returned code: 1 stdout: b'' stderr: b'time="YYYY-MM-DDTHH:MM:SSZ" level=error msg="runc run failed: unable to start container process: error during container init: error mounting \\"/etc/sysconfig/proxy\\" to rootfs at \\"/etc/sysconfig/proxy\\": create mountpoint for /etc/sysconfig/proxy mount: lookup mountpoint target: securejoin.OpenInRoot /etc/sysconfig/proxy: openat2 /etc/sysconfig/proxy: not a directory"\n'
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/init_container.py", line 188, in <module>
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     run(args.spec_dir)
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/init_container.py", line 179, in run
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     execute_init_container_logic(init_container)
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/init_container.py", line 154, in execute_init_container_logic
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     run_init_container_steps(init_container)
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/init_container.py", line 109, in run_init_container_steps
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     run_init_container(init_container, init_container_dir)
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/init_container.py", line 46, in run_init_container
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     run_command([RUNC, 'run', '-b', container_path, container_name])
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX   File "/usr/lib/containerfw/container_util/helper.py", line 38, in run_command
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX     raise Exception('Command failed: stdout %r stderr %r' % (text, err))
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX Exception: Command failed: stdout b'' stderr b'time="YYYY-MM-DDTHH:MM:SSZ" level=error msg="runc run failed: unable to start container process: error during container init: error mounting \\"/etc/sysconfig/proxy\\" to rootfs at \\"/etc/sysconfig/proxy\\": create mountpoint for /etc/sysconfig/proxy mount: lookup mountpoint target: securejoin.OpenInRoot /etc/sysconfig/proxy: openat2 /etc/sysconfig/proxy: not a directory"\n'
YYYY-MM-DDTHH:MM:SS.979Z Wa(03)+ host-XXXXXXX
YYYY-MM-DDTHH:MM:SS.987Z Er(02) host-XXXXXXX <vc-ws1a-broker> Service pre-start command failed with exit code 1.
..
..
YYYY-MM-DDT03:41:25.943Z Er(02) host-XXXXXXX Service batch op START failed. Failed services: 'vc-ws1a-broker'

Manually running the command /usr/bin/runc run -b '/storage/containers/ws1-init-container/xxxxxx' ws1-init-container, gives the same error as seen in the vmon.log.
You are not running any container inside the vCenter(VKS) and not using an external Identity Federation(like Okta and Microsoft Entra ID, etc).

Environment

VMware vCenter Server 8.0.x

Cause

There are two potential cause of the error

1. Proxy missconfigured as a directory and not as file.

2. Potential corruption in the directory /storage/containers/ws1-init-container/xxxxxx.It can be due to missing script, directories or conf files.

Resolution

To resolve the proxy missconfiguration(Cause 1), follow these steps:

Prerequisites

Take a powered-off snapshot of the vCenter Server Appliance.

Steps

Log in to the vCenter Server Appliance shell (SSH) as root.
Validate the file type of the proxy configuration:
```
ls -la /etc/sysconfig/proxy
```
If the output shows a directory (indicated by a d at the start of the permissions string, e.g., drwxr-xr-x), proceed to the next step.
Rename the malformed directory to a backup location:
```
mv /etc/sysconfig/proxy /etc/sysconfig/proxy.bak
```
Recreate the proxy configuration strictly as a standard file using vi:
```
vi /etc/sysconfig/proxy
```

Input the required proxy settings. Copy and paste the following content into the file, then save and exit:

# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="no"

# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY=""

# Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY=""

# Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY=""

# Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY=""

# Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY=""

# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""

# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY=", localhost, 127.0.0.1"

6. Restart the vc-ws1a-broker service:

service-control --start vc-ws1a-broker

If you are not using vc-ws1a-broker service for container or Identity Federations then to address this issue, stop the vc-ws1a-broker service and disable vc-ws1a-broker.

vmon-cli --stop vc-ws1a-broker

vmon-cli --update vc-ws1a-broker --starttype DISABLE

To enable the service is future you can run the below command:

vmon-cli --update vc-ws1a-broker --starttype AUTOMATIC

Additional Information

Stop the vc-ws1a-broker service and disable vc-ws1a-broker if external IDP such as Azure AD and Okta is not used.

vCenter update failing with "Failed to start vc-ws1a-broker" if using proxy