Config drift update on the SDDC manager fails at "DISABLE_CERTIFICATE_REVOCATION_LIST_VALIDATION_IN_NSX-T_DATA_CENTER"
search cancel

Config drift update on the SDDC manager fails at "DISABLE_CERTIFICATE_REVOCATION_LIST_VALIDATION_IN_NSX-T_DATA_CENTER"

book

Article ID: 390253

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • It would be noticed that the config drift update on the SDDC manager when pushing updates to NSX on specific domain/s would fail with the following errors,

2025-03-06T02:47:54.203+0000 DEBUG [vcf_lcm,1a63891795234824,6d2b] [c.v.e.s.l.a.r.c.u.UpgradeAssemblers,http-nio-127.0.0.1-7400-exec-13] Getting the localized name for stage Domain_name_DISABLE_CERTIFICATE_REVOCATION_LIST_VALIDATION_IN_NSX-T_DATA_CENTER
2025-03-06T02:47:54.203+0000 DEBUG [vcf_lcm,1a63891795234824,6d2b] [c.v.e.s.l.a.r.c.u.UpgradeAssemblers,http-nio-127.0.0.1-7400-exec-13] Stage name Domain_name_DISABLE_CERTIFICATE_REVOCATION_LIST_VALIDATION_IN_NSX-T_DATA_CENTER, Parsed staged name DISABLE_CERTIFICATE_REVOCATION_LIST_VALIDATION_IN_NSX-T_DATA_CENTER using domain WLD-Domain
2025-03-06T02:47:54.203+0000 DEBUG [vcf_lcm,1a63891795234824,6d2b] [c.v.e.s.l.c.u.LcmLocalizationTools,http-nio-127.0.0.1-7400-exec-13] Got the class loader java.net.URLClassLoader@5ddfc43d for bundle location at /var/log/vmware/vcf/lcm/thirdparty/bundles/85192dee-1d47-4211-bbdb-999d604f601f/thirdparty/sddcmanager-migration-app/conf/workflows/configdrift

 

Environment

VCF 4.x

Cause

  • The Failures occurs due to password mismatch on the NSX manager's admin account between the SDDC manager and the NSX manager. 
  • When the password status of the NSX manager's admin is validated, it would be found in a disconnected status from the SDDC side.  
  • The migration-app.log at the path : /var/log/vmware/vcf/lcm/thirdparty/bundles/85192dee-1d47-4211-bbdb-999d604f601f/thirdparty/sddcmanager-migration-app/conf/workflows/configdrift/migration-app.log can be checked to get more details on the issue:

c.v.v.migration.helper.UpgradeSwiper, main] Setting upgrade object for upgrade id 55330c87-0065-4042-80ca-348278726875
c.v.v.migration.helper.UpgradeSwiper, main] Status Code: 200 OK for request uri [http://localhost/lcm/upgrades/55330c87-0065-4042-80ca-348278726875] with me
thod [GET]
c.v.v.migration.helper.UpgradeSwiper, main] Operating upgrade element for upgrade id 55330c87-0065-4042-80ca-348278726875
c.v.v.migration.util.RestUtils, main] Status Code: 200 OK for request uri [http://localhost/lcm/upgrades/55330c87-0065-4042-80ca-348278726875] with me
thod [PUT]
c.v.v.migration.helper.UpgradeSwiper, main] Updated the upgrade element for upgrade id 55330c87-0065-4042-80ca-348278726875
com.vmware.vcf.common.nsxt.sd.client.connection.DataPlaneNsxtManagerOperations, pool-5-thread-15] Waiting 900000 ms for NSX manager/cluster to be stable
com.vmware.vcf.common.nsxt.sd.client.connection.SecurityConfigurationServiceImpl, pool-5-thread-15] Security config retrieved {"certificateValidationEnabled":true, "tlsMode":false}
com.vmware.vcf.common.nsxt.sd.client.connection.SecurityConfigurationServiceImpl, pool-5-thread-15] Security config retrieved {"certificateValidationEnabled":true, "tlsMode":false}
com.vmware.vcf.common.nsxt.sd.client.connection.ApiConnection, pool-5-thread-15] Creating apiClient to http://nsx_manager_fqdn:443 with username admin
com.vmware.vcf.common.nsxt.sd.client.connection.ApiConnection, pool-5-thread-15] Created ApiClient connection for nsx_manager_fqdn
com.vmware.vcf.common.nsxt.sd.client.connection.ApiConnection, pool-5-thread-15] Cloned ApiClient connection.
com.vmware.vcf.common.nsxt.sd.helper.NsxtUtils, pool-5-thread-15] Error fetching NSX cluster status
message-[],
data = struct => {error_message=The Credentials were incorrect or the account specified has been locked., error code - 403, module name - common-services},
erroType=UNAUTHERIZED 

com.vmware.vcf.common.nsxt.sd.helper.NsxtUtils$NsxtException: 
        at jdk.internal.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.vmware.vcf.internal.bindings.JavaClassStructConverter.invoke(JavaClassStructConverter.java:79)
        at com.vmware.vcf.internal.bindings.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33)
        at com.vmware.vcf.internal.bindings.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:275)
        at com.vmware.vcf.internal.bindings.TypeConverterImpl.valueToJava(TypeConverterImpl.java:330)
        at com.vmware.vcf.internal.bindings.Stub.convert(Stub.java:454)
        at com.vmware.vcf.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:713)
        at com.vmware.vcf.internal.bindings.Stub.convertError(Stub.java:471)
        at com.vmware.vcf.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:42)
        at com.vmware.vcf.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:21)
        at com.vmware.vcf.internal.protocol.client.rest.DefaultHttpRequestExecutorFactory$DefaultHttpResponseHandler.onResponse(DefaultHttpRequestExecutorFactory.java:80)
        at com.vmware.vcf.internal.protocol.client.rest.apache.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:79)
        at com.vmware.vcf.internal.protocol.client.rest.DefaultHttpRequestExecutorFactory$DefaultHttpRequestExecutor.execute(DefaultHttpRequestExecutorFactory.java:45)
        at com.vmware.vcf.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
        at com.vmware.vcf.internal.bindings.Stub.invoke(Stub.java:247)
        at com.vmware.vcf.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
        at com.vmware.vcf.internal.bindings.Stub.invokeMethod(Stub.java:137)
        at com.vmware.vcf.common.nsxt.gen.status.StatusStub.get(StatusStub.java:46)
        at com.vmware.vcf.common.nsxt.sd.client.connection.ClusterOperations.getStatus(ClusterOperations.java:40)
        at com.vmware.vcf.common.nsxt.sd.client.connection.DataPlaneNsxtManagerOperations.taskWaitClusterStable(DataPlaneNsxtManagerOperations.java:480)
        at com.vmware.vcf.common.fsm.plugins.nsxt.helpers.NsxtUtils.getManagementClusterStatus(NsxtUtils.java:312)
        at com.vmware.vcf.common.fsm.plugins.nsxt.helpers.NsxtCommonOperations.waitForState(NsxtCommonOperations.java:139)
        at com.vmware.vcf.common.fsm.plugins.nsxt.helpers.NsxtUtils.waitForClusterStatusToBeStable(NsxtUtils.java:250)
        at com.vmware.vcf.common.fsm.plugins.nsxt.action.EnableDisableNsxtCrlValidationAction.execute(EnableDisableNsxtCrlValidationAction.java:61)
        at com.vmware.vcf.common.fsm.plugins.nsxt.action.EnableDisableNsxtCrlValidationAction.execute(EnableDisableNsxtCrlValidationAction.java:29)
        at com.vmware.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)
        at com.vmware.sddc.orchestrator.platform.action.RunActionState.invoke(RunActionState.java:62)
        at com.vmware.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:141)
        at com.vmware.sddc.orchestrator.platform.action.RunActionState.invoke(RunActionState.java:62)
        at com.vmware.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:141)
        at com.vmware.sddc.orchestrator.platform.action.RunActionState.invoke(RunActionState.java:62)
        at com.vmware.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:141)
        at com.vmware.sddc.orchestrator.platform.action.RunActionState.invoke(RunActionState.java:62)
        at com.vmware.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.

Resolution

  • To fix the issue, compare the current working password for the admin account with existing password on the NSX manager's admin password.
  • If these are different, the update the NSX manager's admin password to match password listed on the database of the SDDC manager by following document: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/nsxt-dc/3-0/administration-guide/authentication-and-authorization/managing-local-user-accounts/resetting-passwords-on-an-appliance.html
  • Once done, remediate the password on manager using the following steps: 
    • Navigate to security tab
    • Select password manager 
    • From the password list filter for NSX 
    • Click on the 3 dots against the Admin account for the NSX FQDN where the config drift failing. 
    • Click on Remediate -> Provide the same password the password that taken from the SDDC manager's lookup password 
    • You should see the password remediate successfully.  
  • Once the password is Active on the UI, restart the config drift update and we will see the updates going through fine. 
Powered by Wolken Software