vCenter upgrade fails during stage 2 with errors "Failed to Register updatemgr extension" and "Error While configuring vSphere Update Manager firstboot".
search cancel

vCenter upgrade fails during stage 2 with errors "Failed to Register updatemgr extension" and "Error While configuring vSphere Update Manager firstboot".

book

Article ID: 386345

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

VUM fails to register to the vCenter server causing upgrade failure during the upgrade from 7.0u3 to 8.0u3d.

Environment

VMware vCenter Server 7.0

Cause

VUM startup failures were caused by excessive tasks/events, specifically related to Nutanix NCC user login/logout activities. These events overloaded vpxd, which in turn prevented VUM from starting.

Log snippet from firstbootStatus.json

{
             "updatemgr-firstboot": "0:02:33"
         }
     ],
     "failedSteps": "updatemgr-firstboot",
  
updatemgr-firstboot.py_stderr.log
error:
YYYY-MM-DD HH:MM:SS Excption caught in checkException: Register with extension manager
YYYY-MM-DD HH:MM:SS  UpdateMgr received BaseInstallException '{
     "detail": [
         {
             "id": "install.updatemgr.error.extension_reg",
             "translatable": "Failed to register updatemgr extension.",
             "localized": "Failed to register updatemgr extension."
         }
     ],
     "componentKey": "Update Manager",
     "problemId": "install.updatemgr.firstboot",
     "resolution": {
         "id": "install.updatemgr.resolution.common",
         "translatable": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request.",
         "localized": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request."
 
Log snippet from fbInstall.json
 
 
"start_time": "YYYY-MM-DD HH:MM:SS",
     "end_time": "YYYY-MM-DD HH:MM:SS"  >>>>>>>>>>> No time jump seen
 
Log snippet from vmware-vci-vciInstallUtils.log
 
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=VciInstallUtils] Connecting to host "vCenter.FQDN" on port 80 using protocol http
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=VciInstallUtils] VC certificate:
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39879] [Originator@6876 sub=IO.Connection] Attempting connection; <resolver p:0x00007f09d8038170, 'vCenter.FQDN:80', next:<TCP '127.0.0.1 : 80'>>, last e: 0(Success)
YYYY-MM-DD HH:MM:SS warning vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f09d8046bd0, TCP:vCenter.FQDN:80>, (null), duration: 1msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS error vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=IO.Http] User agent failed to send request; (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=VciInstallUtils] Version discovery for 'vim' VMODL from vCenter.FQDN:80 failed: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Udm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAoLRKAEfmRwEITx5saWJ2bW9taS5zbwABYl0eAQFfHgLcWx5saWJ1ZmEtY29tbW9uLnNvAAJrXB4CoYEeAlCHHgKq+zYDwR4CbGlidmNpLXJlZ2lzdHJhci5zbwADEYsCBJnMBXZjaUluc3RhbGxVdGlscwAEbwcGBNspBwUKfwJsaWJjLnNvLjYABCnQAw==[/context] caused by: context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS error vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Registrar] [registrar 1104] Failed to get certificate credentials with vc thumbprint. Unexpected error: Dynamic exception type: Vmomi::Fault::HostNotReachable::Exception
 --> std::exception::what: Fault cause: vmodl.fault.HostNotReachable
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Extension] Connecting to host "vCenter.FQDN" on port 80 using protocol http
YYYY-MM-DD HH:MM:SSverbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Extension] VC certificate:
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=IO.Connection] Attempting connection; <resolver p:0x00007f09d803c510, 'vCenter.FQDN:80', next:<TCP '127.0.0.1 : 80'>>, last e: 0(Success)
YYYY-MM-DD HH:MM:SSwarning vmware-vci-vciInstallUtils[39878] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f09d80468a0, TCP:vCenter.FQDN:80>, (null), duration: 0msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39878] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS error vmware-vci-vciInstallUtils[39878] [Originator@6876 sub=IO.Http] User agent failed to send request; (null), N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39874] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 9, idle: 1
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[46010] [Originator@6876 sub=ThreadPool] Entering worker thread loop
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Extension] Version discovery for 'vim' VMODL from vCenter.FQDN:80 failed: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Udm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAoLRKAEfmRwEITx5saWJ2bW9taS5zbwABYl0eAQFfHgLcWx5saWJ1ZmEtY29tbW9uLnNvAAJrXB4CoYEeAlCHHgKq+zYDYUECbGlidmNpLXJlZ2lzdHJhci5zbwADU4sCBJnMBXZjaUluc3RhbGxVdGlscwAEbwcGBNspBwUKfwJsaWJjLnNvLjYABCnQAw==[/context]
 --> caused by: [context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS error vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Registrar] [registrar 1374] HostNotReachable error: http://vCenter.FQDN:80
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Default] [installerRunVCCommand 701]  [VCSERVER] RegisterExtension returned:-1
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Default] [installerRunVCCommand 402]  [VCSERVER] returning code = 199 from vc check.
 
envoy-access.log

YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:38652 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 603 437 gzip 1 1 0 XX.XX.XXX.XXX:53953 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51712 HTTP/2 - 127.0.0.1:8085 - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:53636 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 571 17059 gzip 7 3 4 XX.XX.XXX.XXX:39932 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51698 HTTP/2 - 127.0.0.1:8085 - "WaitForUpdatesEx"
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:39264 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 571 1033 gzip 2371 2368 0 XX.XX.XXX.XXX:43518 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:47374 HTTP/2 - 127.0.0.1:8085 - "WaitForUpdatesEx"
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 571 10454 gzip 7 4 3 XX.XX.XXX.XXX:43554 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:47374 HTTP/2 - 127.0.0.1:8085 - "WaitForUpdatesEx"
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 667 3588 gzip 3 2 0 XX.XX.XXX.XXX:53953 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51712 HTTP/2 - 127.0.0.1:8085 - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:39794 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 571 1036 gzip 1 1 0 XX.XX.XXX.XXX:43614 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51698 HTTP/2 - 127.0.0.1:8085 - "WaitForUpdatesEx"
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:43378 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -

 

Resolution

  1. Take an *Offline Snapshot* of the vCenter before proceeding to the maintenance window.
  2. Follow the guidance in the below KB to prune all event data to 0 days on the source appliance.
    https://knowledge.broadcom.com/external/article/313506/delete-old-tasks-events-and-statistics-d.html#run_script_for_vPostgres_for_vcenter_appliances

    Example using /tmp directory:
    /opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=0 -v EventMaxAgeInDays=0 -v StatMaxAgeInDays=0 -d VCDB -t -q -f /tmp/2110031_Postgres_task_event_stat_reset_event_sequence.sql
  3. Begin the upgrade by deploying the 8.0.3 appliance during Stage 1. Wait until Stage 1 is Completed and the appliance is deployed 
  4. Open a console to the newly deployed appliance and login to the DCUI via F2
  5. Select Troubleshooting Mode Options
  6. Select Enable BASH Shell and set it to enabled
  7. Select Enable SSH and set it to enabled
  8. scp/upload the following 2 files to the /tmp directory config.yaml and sdkTunnel.json
  9. Replace the files from /tmp to overwrite the config (These two files are attached to the KB) 
    cp /tmp/config.yaml /etc/vmware-envoy/config.yaml
cp /tmp/sdkTunnel.json /etc/vmware-rhttpproxy/endpoints.conf.d/sdkTunnel.json
  10. Begin Stage 2 and proceed with the upgrade.
    Once the upgrade is completed disable the Burst Filter whitelist.

Additional Information

The subsequent review of the vpxd journal shows that the burst filter has been configured but there are no event bursts for either UserLoginSessionEvent/UserLogoutSessionEvent.

By enabling the whitelist, the burst filter will be disabled for these event types which is counterintuitive to the issue raised.

Reference: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-13D8C676-A7B9-4191-8528-9A5FCE4A5494.html 

Configure a key-value to enable the config.vpxd.event.burstFilter.whiteList option.

You can configure the burst filter to compress all types of events except the events that are on the allow list. This parameter allows you to separate event types. It can avoid monitoring a few event types by the burst filter. The default value isvim.event.UserLoginSessionEvent;vim.event.UserLogoutSessionEvent <<< Remove this.

Restart the vpxd service to pick up the new change.

Attachments

config.yaml.gz get_app
sdkTunnel.json.gz get_app
Powered by Wolken Software