Portgroups not populated when adding workload network on Supervisor with Foundation Load Balancer
search cancel

Portgroups not populated when adding workload network on Supervisor with Foundation Load Balancer

book

Article ID: 434081

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • Portgroups do not populate when adding a workload network on a Supervisor configured with a Foundation Load Balancer (FLB).

  • Navigating to Supervisor > Configure > Network > Workload Network > ADD displays an empty table with no Port Groups listed.


  • The wcpsvc.log file shows errors suggesting Internal Server Errors received from the endpoint : 

    YYYY-MM-DDTHH:12:15.750Z warning wcp [vcrestlib/helper.go:176] [opID=wcp-AuthzFilter] Request to service failed; POST, url: http://localhost:1080/rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges, Code: 500, Body: '{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.InternalServerError"],"default_message":"Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError","id":"vapi.bindings.method.impl.unexpected"}]}}'
YYYY-MM-DDTHH:12:15.750Z error wcp [namespace/authz.go:223] [opID=wcp-AuthzFilter] Failed to check privileges for user:  VSPHERE.LOCAL\Administrator, groupnames: [[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]], resources: [{Folder group-v####}], privileges: [System.Read]: HTTP request failed; POST, url: http://localhost:1080/rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges, code: 500, body: '{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.InternalServerError"],"default_message":"Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError","id":"vapi.bindings.method.impl.unexpected"}]}}'

  • The endpoint-access.log files will show entries such as below suggesting the same internal server errors.  : 

    YYYY-MM-DDTHH:12:15.749Z | ####4a57-973b-b0d6#### | 127.0.0.1 | :37468 | "POST /rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges HTTP/1.1" | 500 | 361 | "wcpsvc govmomi/0.48.1 (go1.23.4 X:boringcrypto;linux;amd64)" | - | 788 | com.vmware.cis.authz.privilege:batch_has_privileges | 788 | http://localhost:1080/invsvc/vapi | wcp-###-###-###@VSPHERE.LOCAL | e#####c | com.vmware.vapi.std.errors.internal_server_error
YYYY-MM-DDTHH:20:15.485Z | ####0dd4-9042-a8c1#### | 127.0.0.1 | :40534 | "POST /rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges HTTP/1.1" | 500 | 361 | "wcpsvc govmomi/0.48.1 (go1.23.4 X:boringcrypto;linux;amd64)" | - | 870 | com.vmware.cis.authz.privilege:batch_has_privileges | 870 | http://localhost:1080/invsvc/vapi | wcp-###-###-###@VSPHERE.LOCAL | e#####c | com.vmware.vapi.std.errors.internal_server_error

  • The vmdird.log file shows that there are failing logins to the user "vpxd-svcs-user-###-###-###" : 

    YYYY-MM-DDTHH:33:59.556Z:t@140000000000000:ERROR: VdirPasswordFailEvent from user(cn=vpxd-svcs-user-###-###-###,cn=serviceprincipals,dc=vsphere,dc=local), error(0)()
YYYY-MM-DDTHH:33:59.556Z:t@140000000000000:ERROR: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
YYYY-MM-DDTHH:33:59.556Z:t@140000000000000:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "CN=vpxd-svcs-user-###-###-###,cn=ServicePrincipals,dc=vsphere,dc=local", Method: SASL
YYYY-MM-DDTHH:33:59.591Z:t@140000000000000:ERROR: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

  • The vmware-identity-sts.log logs confirm that the user "vpxd-svcs-user-###-###-###" has been locked  : 

    YYYY-MM-DDTHH:35:10.662Z INFO sts[77:tomcat-http--39] [CorId=####-a989-461c-80bc-####] [com.vmware.identity.diagnostics.VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[vsphere.local], eventid=[USER_NAME_PWD_AUTH_FAILED], level=[ERROR], category=[VMEVENT_CATEGORY_STS], text=[ParameterizedMessage[messagePattern=Failed to authenticate principal [{}]. User account locked., argCount=1, throwableProvided=false]], detailText=[null], corelationId=[####-a989-461c-80bc-####], timestamp=[1773398110662]
YYYY-MM-DDTHH:35:10.662Z ERROR sts[77:tomcat-http--39] [CorId=####-a989-461c-80bc-####] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vpxd-svcs-user-###-###-###@vsphere.local]. User account locked.
YYYY-MM-DDTHH:35:10.662Z INFO sts[77:tomcat-http--39] [CorId=####-a989-461c-80bc-####] [com.vmware.identity.idm.server.IdentityManager] Authentication failed for user [vpxd-svcs-user-###-###-###@vsphere.local] in tenant [vsphere.local] in [15] milliseconds with provider [vsphere.local] of type [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider]
YYYY-MM-DDTHH:35:10.662Z ERROR sts[77:tomcat-http--39] [CorId=####-a989-461c-80bc-####] [com.vmware.identity.idm.server.ServerUtils] Exception 'com.vmware.identity.idm.UserAccountLockedException: User account locked: {Name: vpxd-svcs-user-###-###-###, Domain: vsphere.local}'
com.vmware.identity.idm.UserAccountLockedException: User account locked: {Name: vpxd-svcs-user-###-###-###, Domain: vsphere.local}
        at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.checkUserAccountFlags(VMwareDirectoryProvider.java:1456) ~[libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3162) ~[libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:10191) [libvmware-identity-idm-server.jar:?]
        at com.vmware.identity.idm.client.CasIdmClient.authenticate(CasIdmClient.java:1302) [libvmware-identity-idm-client.jar:?]

Environment

VMware vSphere Kubernetes Service
vSphere Supervisor configured with Foundation Load Balancer

Cause

The vpxd-svcs-user service account is locked in the VMware Directory Service (vmdir), causing authentication and privilege check failures when querying resources for the Workload Network.

Resolution

Use the below steps to recreate the service account password and unlock the user  :

  • Verify the existence of the locked service account by running the following command on the vCenter Server appliance:
    /usr/lib/vmware-vmafd/bin/dir-cli svcaccount list

  • Generate a CIS session token:
    curl -u '[email protected]:<Administrator Password>' -X POST -k https://localhost:443/rest/com/vmware/cis/session

  • Reset the password of the impacted service account using the session token generated in Step 2:
    curl --insecure --request POST --url https://localhost/api//vcenter/svc-account-mgmt/password?action=reset --header 'vmware-api-session-id:<cis session token received in Step 2>' --header 'Content-Type:application/json' --data '{"account_name":"<Service account verified from dir-cli Step 1>"}'

  • Note the new password returned by the command output in the format .
    {"new_password":"<new password>"}

  • Restart the vmware-vpxd-svcs service to apply the changes:
    service-control --restart vmware-vpxd-svcs
Powered by Wolken Software