vCenter Server vpxd-svcs Fails to Start Due to vmdird in Read-Only Mode
search cancel

vCenter Server vpxd-svcs Fails to Start Due to vmdird in Read-Only Mode

book

Article ID: 443407

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter Server services, particularly vpxd and vpxd-svcs, fail to start or remain stopped.
  • The vSphere Client displays a "No Healthy Upstream" error message.
  • Attempts to manually change the VMDir state to Normal using /usr/lib/vmware-vmdir/bin/vdcadmintool are unsuccessful, as the state automatically reverts to Read-Only.
  • Running the fixpsc utility does not resolve the issue.
  • The VDT (vCenter Diagnostic Tool) reports one or more of the following errors:
    • "Nodes registered in multiple SSO sites"
    • "VMDir Local State Check failure"

  • Checking the /var/log/vmware/vmon/vmon.log on the vCenter server service start fails with below :
    YYYY-MM-DDT In(05) host-XXXX Received start request for vpxd-svcs
    YYYY-MM-DDT  In(05) host-XXXX <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 205, in <module>
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:     setup_service_account(VPXD_SVCS_SERVICE_ACCOUNT)
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 586, in setup_service_account
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:     svcaccount.delete_svcaccount(name)
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/svcaccount_prestart_util.py", line 167, in delete_svcaccount
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:     raise InvokeCommandException(error)
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr: cis.exceptions.InvokeCommandException: {
    YYYY-MM-DDT  Wa(03)+ host-XXXX     "detail": [
    YYYY-MM-DDT  Wa(03)+ host-XXXX         {
    YYYY-MM-DDT  Wa(03)+ host-XXXX             "id": "install.ciscommon.command.errinvoke",
    YYYY-MM-DDT  Wa(03)+ host-XXXX             "translatable": "An error occurred while invoking external command : '%(0)s'",
    YYYY-MM-DDT  Wa(03)+ host-XXXX             "args": [
    YYYY-MM-DDT  Wa(03)+ host-XXXX                 "Error 53 while deleting svcaccount \"vpxd-svcs-user-#################\":\ndir-cli failed. Error 53: Possible errors: \nLDAP error: Server is unwilling to perform \nWin Error: Operation failed with error ERROR_BAD_NETPATH (53) \n"
    YYYY-MM-DDT  Wa(03)+ host-XXXX             ],
    YYYY-MM-DDT  Wa(03)+ host-XXXX             "localized": "An error occurred while invoking external command : 'Error 53 while deleting svcaccount \"vpxd-svcs-user-#################\":\ndir-cli failed. Error 53: Possible errors: \nLDAP error: Server is unwilling to perform \nWin Error: Operation failed with error ERROR_BAD_NETPATH(53) \n'"
    YYYY-MM-DDT  Wa(03)+ host-XXXX         }
    YYYY-MM-DDT  Wa(03)+ host-XXXX     ],
    YYYY-MM-DDT  Wa(03)+ host-XXXX     "componentKey": null,
    YYYY-MM-DDT  Wa(03)+ host-XXXX     "problemId": null,
    YYYY-MM-DDT  Wa(03)+ host-XXXX     "resolution": null
    YYYY-MM-DDT  Wa(03)+ host-XXXX }
    YYYY-MM-DDT  Wa(03) host-XXXX <vpxd-svcs> Service pre-start command's stderr:
    YYYY-MM-DDT  Wa(03)+ host-XXXX
    YYYY-MM-DDT  Er(02) host-XXXX <vpxd-svcs> Service pre-start command failed with exit code 1.
    YYYY-MM-DDT Wa(03) host-XXXX [ReadSvcSubStartupData] No startup information from vpxd-svcs

  • Checking the /var/log/vmware/vmdird/vmdird.log records the following directory constraint errors:
    ERROR: VmDirSendLdapResult: Request (Modify), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in read-only mode), (0) socket (###########)
    WARNING: LoginBlocked DN (cn=sps-##############,cn=serviceprincipals,dc=vsphere,dc=local), error (9239)()
    ERROR: Bind Request Failed (########) error 49: Protocol version: 3, Bind DN: "CN=sps-##############,cn=ServicePrincipals,dc=vsphere,dc=local", Method: SASL

Environment

  • vCenter Server 8.0U3

  • Enhanced Linked Mode (ELM)

Cause

Stale directory objects and inconsistencies within the Enhanced Linked Mode (ELM) configuration cause vmdird to enter read-only mode, which prevents the successful creation and deletion of service accounts required by vCenter services.

Resolution

The vpxd-svcs service can be started successfully once the vmdird database state is restored to NORMAL. To achieve this, temporarily separate the vCenter Servers from the Enhanced Linked Mode (ELM) topology, validate the standalone vCenter functionality, remove any stale directory entries, and then re-establish Enhanced Linked Mode.

Warning: The following procedure temporarily breaks Enhanced Linked Mode (ELM) between the vCenter Servers. Before proceeding, take offline snapshots of all vCenter Servers participating in the ELM topology to ensure a consistent rollback point and maintain directory state integrity.

For instructions on taking consistent offline snapshots of ELM environments, refer to Broadcom KB 313886.

To resolve the issue please follow the below steps :

1. Split the Enhanced Linked Mode Topology

To eliminate dependencies between the linked vCenter Servers and allow independent validation of the affected vCenter Server, split the ELM configuration.

Refer to the following KB for details regarding splitting the Enhanced Linked Mode : Broadcom KB 370062

2. Verify vCenter Services

After the ELM split operation completes successfully, verify that all required services are running on each standalone vCenter Server.

service-control --status --all

Confirm that the vmdird, vpxd, and vpxd-svcs services are running and that the vmdird database state remains NORMAL.

3. Validate the VMDir Database for Stale Entries

Once the affected vCenter Server is operating independently, validate the VMDir database using JXplorer and ensure no stale or orphaned entries remain.

Refer to the following KB for using the JXplorer to connect to the vSphere Single Sign on : Broadcom KB 301632

Note: If assistance is required while validating or cleaning up VMDir entries, contact Broadcom Support.

4. Re-establish Enhanced Linked Mode

After confirming that the standalone vCenter Server is healthy and free of stale directory entries, repoint the vCenter Server to re-establish Enhanced Linked Mode.

Refer to the following techdocs that gives information on how to Repoint vCenter Server to Another vCenter Server in a Different Domain

5. Validate Services and Replication Health

After the repoint operation completes:

Verify that all vCenter services are running.

service-control --status --all

Verify directory replication health between the linked vCenter Servers.

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w 'administrator_sso_password'

Ensure that partner replication reports a healthy status and no replication errors are present.

6. Validate vCenter Functionality

Log in to the vSphere Client and verify the following:

->The vCenter UI is accessible.
->Enhanced Linked Mode is visible and functioning correctly.
->Inventory objects are displayed as expected.
->Administrative operations complete successfully.
->No service health alarms or replication issues are reported.

 

 

 

 

 

 

Additional Information

For instructions on how to use fixpsc, refer the Broadcom KB 316566

For instructions on how to use  VCF Diagnostic Tool for vSphere (VDT), refer the Broadcom KB 344917

For instructions on how to Determine replication agreements and status with the PSC , refer the Broadcom KB 316504

Powered by Wolken Software