Failed to Upgrade NSX component on the ESXi hosts from version 4.1.2.3 to 4.2.3.0.
search cancel

Failed to Upgrade NSX component on the ESXi hosts from version 4.1.2.3 to 4.2.3.0.

book

Article ID: 415921

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

•   Host Compliance check failure on vCenter, hosts are out of compliance, VLCM health check fails

•   NSX Upgrade-Coordinator logs show  -- "Health check failed to execute on item 'Workload Management Platform' for service 'domain-c000'. Verify that the service Workload Management Platform is running and try again."

NSX/var/log/vmware/upgrade-coordinator/

2025-10-03T10:02:55.595Z  INFO task-executor-17-1-workitem-HOST-vlcm-c0000003-2800-400e-9000-b00000000008 VLCMOperationsServiceImpl 3000007 FABRIC XXXXXXXXXXXXXXXXXXXX "Host 'esxi.domain.com' was not processed, the reason: 'Health Check for 'Domain-CL' failed'","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX "Health check fails to retrieve data about service 'Workload Management Platform' on 'domain-c000'. Verify that the service 'Workload Management Platform' is running and try again.", "localized":"Health check failed to execute on item 'Workload Management Platform' for service 'domain-c000'. Verify that the service Workload Management Platform is running and try again."

•   We can see from the vum-server logs, that the health check session fails to authorize, with missing privileges on Host.Config.Maintenance

vc/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

---------------> Session fails to authenticate
2025-10-22T15:19:01.549+02:00 info vmware-vum-server[1130769] [Originator@6876 sub=VumVapiPrivilegeChecker opID=10000008-b000-4000-a000-b0000000000] [PrivilegeCheckerImpl 232] Resource:domain-c000, is CLUSTER MoId
2025-10-22T15:19:01.553+02:00 info vmware-vum-server[1130769] [Originator@6876 sub=IntegrityAuthzServiceHelper opID=10000008-b000-4000-a000-b0000000000] [AuthzServiceHelper 62] Entity Priv Result: Entity:domain-c000
2025-10-22T15:19:01.553+02:00 info vmware-vum-server[1130769] [Originator@6876 sub=IntegrityAuthzServiceHelper opID=10000008-b000-4000-a000-b0000000000] [AuthzServiceHelper 69] Entity Priv Result: PrivId:VcIntegrity.lifecycleHealth.Read, PrivValue:true
2025-10-22T15:19:01.553+02:00 info vmware-vum-server[1130769] [Originator@6876 sub=VumVapiAsyncTelemetryFilter opID=10000008-b000-4000-a000-b0000000000] [TelemetryVapiProvider 117] Invoking method com.vmware.esx.health.clusters.status.check
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Started: health check query for [domain-c000], perspective: [BEFORE_ENTER_MAINTENANCE]
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Got all services.
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Disabled checks:
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Excluded checks:
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Filtering health checks in provider [esx]...
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] CheckContext: {entityMoId: "domain-c000", vapiSession: "bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa", env: {"Host part of VMC": false, "vLCM-VMC integration, Pod service enabled": false, }}, ClusterCheckContext: {spec: {{ com.vmware.esx.health.clusters.check_spec : { exclude_checks : [ ] , hosts : Optional< [ host-xx, ] >, maintenance_mode_type : Optional< >, memory_reservation : Optional< >, perspective : BEFORE_ENTER_MAINTENANCE, upgrade_actions : Optional< >, } }} }
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] [domain-c000] A provider [esx] has finished (8 remaining).
2025-10-22T15:19:01.554+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Filtering health checks in provider [vcsa]...
2025-10-22T15:19:01.556+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=vmomi.soapStub[5] opID=10000008-b000-4000-a000-b0000000000] SOAP request returned HTTP failure; <<io_obj p:0x00007fdd5006e270, h:28, <TCP '127.0.0.1 :
 50078'>, <TCP '127.0.0.1 : 1080'>>, /sdk>, method: GetConfigurationEx; code: 500(Internal Server Error); fault: (vim.fault.NotAuthenticated) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    object = 'vim.ClusterComputeResource:domain-c000',
-->    privilegeId = "",
-->    missingPrivileges = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fdd5006e270, h:28, <TCP '127.0.0.1 : 50078'>, <TCP '127.0.0.1 : 1080'>>, /sdk>]: GetConfigurationEx
--> The session is not authenticated."
--> }
2025-10-22T15:19:01.559+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Getting 'delegable' SSO token
2025-10-22T15:19:01.559+02:00 info vmware-vum-server[1139538] [Originator@6876 sub=EHP opID=10000008-b000-4000-a000-b0000000000] Acquiring SAML token with extension certificate...
 
 

-----------> Unauthorized
2025-10-22T15:19:01.731+02:00 info vmware-vum-server[1139546] [Originator@6876 sub=SamlAsyncApiProvider opID=10000008-b000-4000-a000-b0000000000] [SamlAsyncApiProvider 86] Creating vAPI session
2025-10-22T15:19:01.864+02:00 info vmware-vum-server[1136721] [Originator@6876 sub=SamlAsyncApiProvider opID=10000008-b000-4000-a000-b0000000000] [SamlAsyncApiProvider 92] Successfully created vAPI session
2025-10-22T15:19:01.875+02:00 warning vmware-vum-server[1139546] [Originator@6876 sub=ClusterOps opID=10000008-b000-4000-a000-b0000000000] [ClusterOps 874] Failed to create maintenance requests. Reason: Error: Error:
-->    com.vmware.vapi.std.errors.unauthorized
--> Messages:
-->    com.vmware.api.vcenter.host.maintenance.requests.auth.no.priv<Session missing privilege: 'object: host-xx:c1e3fff2-0ac6-463c-ac7f-d45203422826 privileges: Host.Config.Maintenance'>
--> . Retry after 10 seconds

•   vpxd shows the vpxd-extension-xxxxxxx User data can't be added to scheduler.

vc/var/log/vmware/vmware-vpxd/vpxd.log

2025-10-22T15:19:01.600+02:00 info vpxd[1110784] [Originator@6876 sub=vpxLro opID=f1xxxxxx] [VpxLRO] -- BEGIN lro-8645 -- SessionManager -- vim.SessionManager.loginByToken -- 521b3514-1e87-1598-693e-1f9d0e043cd9
2025-10-22T15:19:01.603+02:00 info vpxd[1110784] [Originator@6876 sub=UserDirectorySso opID=f1xxxxxx] GetUserInfoInternal(vsphere.local\vpxd-extension-cxxxxxxx-2xxx-4xxx-9xxx-bxxxxxxxxxxx, false) res: VSPHERE.LOCAL\vpxd-extension-cxxxxxxx-2xxx-4xxx-9xxx-bxxxxxxxxxxx
2025-10-22T15:19:01.603+02:00 info vpxd[1110784] [Originator@6876 sub=UserDirectorySso opID=f1xxxxxx] GetUserInfoInternal(VSPHERE.LOCAL\ServiceProviderUsers, true) res: VSPHERE.LOCAL\ServiceProviderUsers
2025-10-22T15:19:01.603+02:00 warning vpxd[1110784] [Originator@6876 sub=AuthorizeManager opID=f1xxxxxx] Refresh function is not configured.User data can't be added to scheduler.User name: VSPHERE.LOCAL\vpxd-extension-cxxxxxxx-2xxx-4xxx-9xxx-bxxxxxxxxxxx
2025-10-22T15:19:01.603+02:00 info vpxd[1110784] [Originator@6876 sub=User opID=f1xxxxxx] Login token: SamlToken [subject={Name: vpxd-extension-cxxxxxxx-2xxx-4xxx-9xxx-bxxxxxxxxxxx; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: AnalyticsService.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ServiceProviderUsers; Domain:vsphere.local}, {Name: vStatsGroup; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2025-10-22 13:19:01.570, endTime=2025-10-22 14:19:01.570, renewCount=0, delegableCount=0, isSolution=true, type=Saml_HOK]
2025-10-22T15:19:01.605+02:00 info vpxd[1110784] [Originator@6876 sub=vpxLro opID=f1xxxxxx] [VpxLRO] -- FINISH lro-8645

 

 

Environment

vCenter 8.x

ESXi 8.x

Cause

This is due to the vpxd-extension is under vStatsGroup, and this group role does not have Host.Config.Maintenance under the vStatsAdmin role permissions.

Resolution

Resolution:

1. vCenter > Administration > Access Control > Roles

2. Select vStatsAdmin, and click on Edit

3. Select Host > Configurations > Maintenance, and click Save

4. Retry the Image Compliance check.

Workaround:

•    Make the Host standalone (Not inside any cluster), perform the NSX component Upgrade on the standalone host. This will complete successfully, because this will allow Host object gets the direct permission and not group permissions when outside the cluster.

•    Create a new cluster and add the Host with upgraded components and create the Image Custer under Updates, and remediate other hosts by moving them into the new host.

Additional Information

* Creating and assigning a role with privileges to create and manage virtual machine to a Domain or Local User/Group
https://knowledge.broadcom.com/external/article/316586/creating-and-assigning-a-role-with-privi.html

* Remediation of host fails with health check failure
https://knowledge.broadcom.com/external/article?articleNumber=404575

Powered by Wolken Software