vCenter upgrade fails during stage 2 with errors "Failed to Register updatemgr extension" and "Error While configuring vSphere Update Manager firstboot".
search cancel

vCenter upgrade fails during stage 2 with errors "Failed to Register updatemgr extension" and "Error While configuring vSphere Update Manager firstboot".

book

Article ID: 386345

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • VUM fails to register to the vCenter server causing upgrade failure during the upgrade from 7.0u3 to 8.0u3d.
  • firstbootStatus.json reports below error

{
             "updatemgr-firstboot": "0:02:33"
         }
     ],
     "failedSteps": "updatemgr-firstboot",
  
updatemgr-firstboot.py_stderr.log
error:
YYYY-MM-DD HH:MM:SS Excption caught in checkException: Register with extension manager
YYYY-MM-DD HH:MM:SS  UpdateMgr received BaseInstallException '{
     "detail": [
         {
             "id": "install.updatemgr.error.extension_reg",
             "translatable": "Failed to register updatemgr extension.",
             "localized": "Failed to register updatemgr extension."
         }
     ],
     "componentKey": "Update Manager",
     "problemId": "install.updatemgr.firstboot",
     "resolution": {
         "id": "install.updatemgr.resolution.common",
         "translatable": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request.",
         "localized": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request."

  • fbInstall.json reports below entries.

"start_time": "YYYY-MM-DD HH:MM:SS",
     "end_time": "YYYY-MM-DD HH:MM:SS"  >>>>>>>>>>> No time jump seen

  • vmware-vci-vciInstallUtils.log reports below error

YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39879] [Originator@6876 sub=IO.Connection] Attempting connection; <resolver p:0x00007f09d8038170, 'vCenter.FQDN:80', next:<TCP '127.0.0.1 : 80'>>, last e: 0(Success)
YYYY-MM-DD HH:MM:SS warning vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f09d8046bd0, TCP:vCenter.FQDN:80>, (null), duration: 1msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]

[context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=VciInstallUtils] Version discovery for 'vim' VMODL from vCenter.FQDN:80 failed: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
[context]zKq7AVECAQAAAP////8Udm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAoLRKAEfmRwEITx5saWJ2bW9taS5zbwABYl0eAQFfHgLcWx5saWJ1ZmEtY29tbW9uLnNvAAJrXB4CoYEeAlCHHgKq+zYDwR4CbGlidmNpLXJlZ2lzdHJhci5zbwADEYsCBJnMBXZjaUluc3RhbGxVdGlscwAEbwcGBNspBwUKfwJsaWJjLnNvLjYABCnQAw==[/context] caused by: context]zKq7AVECAQAAAP////8Odm13YXJlLXZjaS12Y2lJbnN0YWxsVXRpbHMAADMcU2xpYnZtYWNvcmUuc28AAP4XQgAfP0MAjJlKACwSSACebUgAZuZCAA1lNwDlZjcABOw3ABdFOAC7D1EBsI4AbGlicHRocmVhZC5zby4wAALf+g9saWJjLnNvLjYA[/context]
YYYY-MM-DD HH:MM:SS error vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Registrar] [registrar 1104] Failed to get certificate credentials with vc thumbprint. Unexpected error: Dynamic exception type: Vmomi::Fault::HostNotReachable::Exception
 --> std::exception::what: Fault cause: vmodl.fault.HostNotReachable
YYYY-MM-DD HH:MM:SS info vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Extension] Connecting to host "vCenter.FQDN" on port 80 using protocol http
YYYY-MM-DD HH:MM:SSverbose vmware-vci-vciInstallUtils[39864] [Originator@6876 sub=Extension] VC certificate:
YYYY-MM-DD HH:MM:SS verbose vmware-vci-vciInstallUtils[39881] [Originator@6876 sub=IO.Connection] Attempting connection; <resolver p:0x00007f09d803c510, 'vCenter.FQDN:80', next:<TCP '127.0.0.1 : 80'>>, last e: 0(Success)
YYYY-MM-DD HH:MM:SSwarning vmware-vci-vciInstallUtils[39878] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007f09d80468a0, TCP:vCenter.FQDN:80>, (null), duration: 0msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

  • envoy-access.log reports below errors

YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:38652 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 603 437 gzip 1 1 0 XX.XX.XXX.XXX:53953 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51712 HTTP/2 - 127.0.0.1:8085 - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS CONNECT /sdkTunnel 503 upstream_reset_before_response_started{overflow} UO 0 81 - 0 - - XX.XX.XXX.XXX:53636 HTTP/1.1 - XX.XX.XXX.XXX:80 - - - - - -
YYYY-MM-DD HH:MM:SS info envoy[19499] [Originator@6876 sub=Default] YYYY-MM-DD HH:MM:SS POST /sdk 200 via_upstream - 571 17059 gzip 7 3 4 XX.XX.XXX.XXX:39932 HTTP/1.1 TLSv1.2 XX.XX.XXX.XXX:443 127.0.0.1:51698 HTTP/2 - 127.0.0.1:8085 - "WaitForUpdatesEx"

 

Environment

VMware vCenter Server 7.0

VMware vCenter Server 8.0

Cause

VUM startup failures were caused by excessive tasks/events, specifically related to Nutanix NCC user login/logout activities. These events overloaded vpxd, which in turn prevented VUM from starting.

Resolution

  1. Take an *Offline Snapshot* of the vCenter before proceeding to the maintenance window.
  2. Follow the guidance in the below KB to prune all event data to 0 days on the source appliance.
    Delete old tasks, events and statistics data in vCenter Server 5.x, 6.x, 7.x and 8.x

    Example using /tmp directory:

/opt/vmware/vpostgres/current/bin/psql -U postgres -v TaskMaxAgeInDays=0 -v EventMaxAgeInDays=0 -v StatMaxAgeInDays=0 -d VCDB -t -q -f /tmp/2110031_Postgres_task_event_stat_reset_event_sequence.sql

     3.Begin the upgrade by deploying the 8.0.3 appliance during Stage 1. Wait until Stage 1 is Completed and the appliance is deployed 

     4. Open a console to the newly deployed appliance and login to the DCUI via F2

     5. Select Troubleshooting Mode Options

     6. Select Enable BASH Shell and set it to enabled

     7. Select Enable SSH and set it to enabled

     8. scp/upload the following 2 files to the /tmp directory config.yaml and sdkTunnel.json

     9.Replace the files from /tmp to overwrite the config (These two files are attached to the KB)

cp /tmp/config.yaml /etc/vmware-envoy/config.yaml
cp /tmp/sdkTunnel.json /etc/vmware-rhttpproxy/endpoints.conf.d/sdkTunnel.json

   10. Begin Stage 2 and proceed with the upgrade.
         Once the upgrade is completed disable the Burst Filter whitelist.

Additional Information

The subsequent review of the vpxd journal shows that the burst filter has been configured but there are no event bursts for either UserLoginSessionEvent/UserLogoutSessionEvent.

By enabling the whitelist, the burst filter will be disabled for these event types which is counterintuitive to the issue raised.

Reference: vSphere Monitoring and Performance

Configure a key-value to enable the config.vpxd.event.burstFilter.whiteList option.

Further information: Configure Events Burst Filter

You can configure the burst filter to compress all types of events except the events that are on the allow list. This parameter allows you to separate event types. It can avoid monitoring a few event types by the burst filter. The default value isvim.event.UserLoginSessionEvent;vim.event.UserLogoutSessionEvent <<< Remove this.

Restart the vpxd service to pick up the new change.

Attachments

config.yaml.gz get_app
sdkTunnel.json.gz get_app
Powered by Wolken Software