During update to 7.0 U3+, envoy service fails to start with error initializing configuration '/etc/vmware-envoy/config.yaml'

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
During an update from 7.0 to 7.0 U3 or higher, the update may fail due to envoy service not starting.

In the /var/log/vmware/applmgmt/PatchRunner.log you see the following:

 stderr: Error executing start on service envoy. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "envoy"
            ],
            "localized": "An error occurred while starting service 'envoy'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

In the /var/log/vmware/envoy/envoy-##.log you see the following:

2022-08-19T18:23:41.253Z Section for VMware VirtualCenter, pid=38294, version=7.0.3, build=20146909, option=Release
2022-08-19T14:23:41.284Z critical envoy[38289] [Originator@6876 sub=main] error initializing configuration '/etc/vmware-envoy/config.yaml': Unable to deserialize during wireCast()
Unable to deserialize during wireCast()

In  the /var/log/vmware/applmgmt/software-packages.log you see the following:

warning: /etc/vmware-envoy/config.cfg created as /etc/vmware-envoy/config.cfg.rpmnew

Environment

VMware vCenter Server 7.0.2
VMware vCenter Server 7.0.3
VMware vCenter Server 7.0.0

Cause

This is caused by the RPM install process not overwriting the original config.cfg file in /etc/vmware-envoy/ due to it being changed in the past.

Checking /etc/vmware-envoy/ you see 2 config.cfg* files (one is config.cfg and another is config.cfg.rpmnew)

The service fails to start because the config.cfg file is missing the following line, which the new binary requires:
--bootstrap-version 3

Resolution

Currently there is no resolution.

 

Workaround:
Since the update failed, the appliance will need to be reverted to previous snapshot or backup.  
Take a new backup/snapshot and then perform ONE of the following steps.

Replace the config.cfg file in /etc/vmware-envoy/ with the file attached to this KB. 

• Copy config.cfg.rpmnew using WinSCP to the host into /tmp
• On host run the command to backup existing config: cp /etc/vmware-envoy/config.cfg /tmp/config.cfg.bak
• Next run this to copy the new file over the existing: cp /tmp/config.cfg.rpmnew /etc/vmware-envoy/config.cfg

OR Update the config.cfg file (using a text editor like vi) in /etc/vmware-envoy/ to look like:

#!/bin/sh

ACCESS_LOG_DIRECTORY=/var/log/vmware/envoy # controls where rolling access log files are stored
ACCESS_LOG_MAXFILENUM=10 # controls number of access log files
ACCESS_LOG_MAXFILESIZE=52428800 # controls access log file size

SERVICE_LOG_LEVEL=info # controls level of service logger
SERVICE_LOG_FORMAT="%Y-%m-%dT%H:%M:%S.%eZ %l envoy[%t] [Originator@6876 sub=%n] %v"
SERVICE_LOG_DIRECTORY=/var/log/vmware/envoy # controls where rolling service log files are stored
SERVICE_LOG_MAXFILENUM=5 # controls number of service log files
SERVICE_LOG_MAXFILESIZE=5242880 # controls service log file size

SERVICE_DEFAULT_ARGS=(
-c /etc/vmware-envoy/config.yaml
-l $SERVICE_LOG_LEVEL
--log-format "$SERVICE_LOG_FORMAT"
--service-cluster vmw_cluster
--drain-time-s 60
--disable-hot-restart
--bootstrap-version 3
)

*Note: The new configuration needed for the updated version is the --bootstrap-version 3 at the bottom of the file.

Once the file is updated on the VCSA, you can retry the update.