Cannot Delete Service Deployment - Agency (for HostBased VMs) delete failure on cluster.

Products

• VMware NSX

Issue/Introduction

In NSX GUI:

• Agency(HostBased VMs) delete fails with: "Agency null not found" error code 4023.

 

 

On vCenter Server, in the /var/log/vmware/eam/eam.log file, similar messages to the following are found:

22023-09-28T13:39:54.946Z | INFO | vlsi | SessionManager.java | 274 | Logged in user using cookie: <redacted> to vCenter. Logical user session ID: <redacted> with physical session cookie: <redacted>

2023-09-28T13:39:54.947Z | INFO | vlsi | ClientAuthenticator.java | 254 | User logged in: com.vmware.nsx.management.nsxt.<UUID-redacted> (version eam/6.7)

2023-09-28T13:39:54.947Z | INFO | vlsi | VcEventManager.java | 792 | [EventIndex: 88] Posting event.

2023-09-28T13:39:54.951Z | INFO | vim-async-0 | VcEventManager.java | 806 | [EventIndex: 88] Event posted.

2023-09-28T13:39:54.948Z | ERROR | vlsi | ClientAuthenticator.java | 368 | Validation failed

com.vmware.eam.security.NotAuthorized: com.vmware.nsx.management.nsxt.<UUID-redacted>

        at com.vmware.eam.security.ClientAuthenticator.authorize(ClientAuthenticator.java:353) ~[eam-server.jar:?]

        at com.vmware.eam.security.ClientAuthenticator.validate(ClientAuthenticator.java:241) [eam-server.jar:?]

        at com.vmware.eam.security.ClientAuthenticator.validate(ClientAuthenticator.java:219) [eam-server.jar:?]

        at com.vmware.vim.vmomi.server.impl.ValidationStartTask.run(ValidationStartTask.java:69) [vlsi-server.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]

2023-09-28T13:39:54.952Z | INFO | vlsi | VcEventManager.java | 792 | [EventIndex: 89] Posting event.

2023-09-28T13:39:54.958Z | INFO | vim-async-0 | VcEventManager.java | 806 | [EventIndex: 89] Event posted.

2023-09-28T13:39:54.961Z | ERROR | vlsi | LocalizationFilter.java | 94 | API ERROR: HostVMAgency(ID:'Agency:<UUID-redacted>:null').getRuntime[opId=239072463, sessionId=<redacted>]. Fault:

eam.fault.NotAuthorized {

}

(eam.fault.NotAuthorized) {

   faultCause = null,

   faultMessage = null

}

        at com.vmware.eam.security.ClientAuthenticator.setFault(ClientAuthenticator.java:378)

        at com.vmware.eam.security.ClientAuthenticator.validate(ClientAuthenticator.java:225)

        at com.vmware.vim.vmomi.server.impl.ValidationStartTask.run(ValidationStartTask.java:69)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

 

2023-09-28T13:39:54.975Z | INFO | vim-async-1 | OpIdLogger.java | 35 | [PropertyCollector(session[<UUID-redacted>]<UUID-redacted>)->WaitForUpdatesEx:d5fe93eb7df967a8] Completed.

2023-09-28T13:39:54.976Z | INFO | vim-inv-update | ClientAuthenticator.java | 445 | User logged out of vCenter: com.vmware.nsx.management.nsxt.<UUID-redacted>

2023-09-28T13:39:54.976Z | INFO | vim-inv-update | VcEventManager.java | 792 | [EventIndex: 90] Posting event.

 

Environment

VMware NSX

Cause

The main cause of this NotAuthorized exception is due to multi-nsx being disabled. The multi-nsx is on for Compute Managers <UUID-redacted>, but later it is disabled (see the below logs) :

 

 multi_nsx enabled:

nsx_manager#/var/log/cm-inventory/cm-inventory.6.log:2023-07-24T17:47:45.554Z INFO org.corfudb.runtime.collections.streaming.StreamPollingScheduler-worker-0 CmPluginManagerImpl 25328 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Compute manager update completed CmRegistrationConfig [id=<UUID-redacted>, server=#####.#####.LOCAL, cmType=vCenter, credential=VerifiableAsymmetricLoginCredential [asymmetricCredential=###, credentialKey=###, credentialVerifier=###, hashCode()=##########parentProperties=LoginCredential [credentialType=VerifiableAsymmetricLoginCredential]], force=false, cmProperties={create_service_account=true, is_multi_nsx=true, is_vc_link=false}, reverseProxyHttpsPort=443, reverseProxyHttpPort=80,cmRoleId=-#########,domain=null, hashCode()=##########]

 

 mult_nsx disabled:

nsx_manager#/var/log/cm-inventory/cm-inventory.5.log:2023-07-26T02:00:15.534Z INFO org.corfudb.runtime.collections.streaming.StreamPollingScheduler-worker-1 CmPluginManagerImpl 25328 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Compute manager update completed CmRegistrationConfig [id=<UUID-redacted>, server=#####.#####.LOCAL, cmType=vCenter, credential=VerifiableAsymmetricLoginCredential [asymmetricCredential=###, credentialKey=###, credentialVerifier=###, hashCode()=#########parentProperties=LoginCredential [credentialType=VerifiableAsymmetricLoginCredential]], force=false, cmProperties={create_service_account=true, is_multi_nsx=false, is_vc_link=false}, reverseProxyHttpsPort=443, reverseProxyHttpPort=80,cmRoleId=#########,domain=null, hashCode()=#########]

Resolution

Issue is present in all NSX versions where MultiNSX feature is present.

Workaround:

• Enable/Disable MultiNSX feature in NSX Compute Manager settings and try deleting Agency configuration again.

 

Additional Information

Impact/Risks:

Agency delete will fail on cluster from NSX. ESXi hosts will fail to vMotion and power on VMs.