vCenter PNID Change fails with "Failed to regenerate certificates"
search cancel

vCenter PNID Change fails with "Failed to regenerate certificates"

book

Article ID: 324594

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • PNID change fails with "Failed to regenerate certificates" error
  • In the /var/log/vmware/applmgmt/pnid_change.log, you will find entries similar to:
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-run_command():81 - INFO - Running command: /usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/tmp/tmpxe4l1fu4 --cert=/tmp/tmpanpyys4f --config=/var/vmware/applmgmt/pnid/pnid_cert.cfg
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-run_command():99 - INFO - Command exited with exitcode : 0
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-run_command():109 - INFO - Done running command
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-generate_ssl_cert():782 - INFO - Successfully updated machine ssl certificates in vecs
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-restart_services():1560 - ERROR - unidentifiable C++ exception
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-update_task_status():1419 - INFO - Task : Failed to regenerate certificates.

 

  • In the vmdir logs at /var/log/vmware/vmdird/vmdird-syslog.log, you will find entries similar to:
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139645182822208: VmDir State (3)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139645182822208: Lotus Vmdird: running... state (3)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139645182822208: Lotus Vmdird: running in FIPS mode.
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623061772032: Vmkdc: initializing directory
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623061772032: Vmkdc: VmKdcdStateSet(1)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623162386176: VmDirUpdateDCNameToLocalNode - PNID: (vcenter.vclass2.local.vclass.local)
[YYYY-MM-DDTHH:MM:SS] info vmdird  Starting VMware Directory Servicedone
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623162386176: VmDirUpdateDCNameToLocalNode - pszDCName: (vcenter.vclass2.local.vclass.local)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623162386176: Successfully notified VMAFD to update DC Name to local node
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623078557440: VmDirUpdateDCNameToLocalNode - PNID: (vcenter.vclass2.local.vclass.local)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623078557440: VmDirUpdateDCNameToLocalNode - pszDCName: (vcenter.vclass2.local.vclass.local)
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@139623078557440: Successfully notified VMAFD to update DC Name to local node
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139623053379328: VmDirSRPGetIdentityData ([email protected]) failed, (9611)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139623053379328: VmDirSRPGetIdentityData ([email protected]) failed, (9611)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139623053379328: SASLSessionStart: sasl error (-20)(SASL(-13): user not found: no secret in database)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139623053379328: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL start failed.)), (0) socket (127.0.0.1)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139623053379328: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "", Method: SASL

 

 

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.x

Cause

  • The new PNID, which gets updated as part of pnid change workflow, will be incorrect. Certificate regeneration using the incorrect PNID fails and causes this issue.
  • This issue occurs if we are performing a pnid change where the DC Account name is the shortname of the PNID.
  • In the /var/log/vmware/applmgmt/pnid_change.log, you will find below entries using which we can obtain the DC Account name, Old PNID, and new PNID
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-check_isUseradministrator():2175 - INFO - DC Account name : vcenter is not same as PNID : vcenter.vclass.local
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-prepare_pnid_change():1721 - INFO - Old PNID : vcenter.vclass.local
[YYYY-MM-DDTHH:MM:SS] - pnid_utils-prepare_pnid_change():1724 - INFO - Starting PNID Change to vcenter.vclass2.local with task id : 9fb468a3-36f9-45cc-8338-627cfe20d924:com.vmware.appliance.networking

         From the above log snippets, we have the following:

DC Account name: vcenter
Old PNID:        vcenter.vclass.local
New PNID:        vcenter.vclass2.local

  • With the above parameters, the pnid change workflow will update the new pnid as vcenter.vclass2.local.vclass.local, which is incorrect

Resolution

This is a known issue with the PNID change workflow, where it does not update the new PNID correctly for this specific use case.

Workaround:
To workaround this issue, we need to update the DC Account name to the current pnid of the vCenter.

  • Roll back the vCenter to a state before the PNID change
  • Run vmafd-cli change-pnid with the current pnid of the vCenter (Ensure vcenter.vclass.local is resolvable)

/usr/lib/vmware-vmafd/bin/vmafd-cli change-pnid --pnid vcenter.vclass.local --user-name 'administrator' --password <>

  • Restart all services and check functionality
  • With a valid snapshot or backup of the vCenter in place, perform the PNID change from the VAMI portal



Powered by Wolken Software