Host commissioning and decommissioning operations fail in SDDC Manager with error "DAY_N_GUARDRAILS Check Type HOST_COMMISSION/HOST_DECOMMISSION Status Is FAIL"

Products

• VMware SDDC Manager

Issue/Introduction

Scenario 1

Commissioning Hosts in SDDC Manager fails with below Error, 

HOST_COMMISSION operation validation failed due to: HOST_COMMISSION operation cannot be performed on resource ########## as these checks have failed Failed to evaluate lambda class LambdaInputRest { checkSetType: DAY_N_GUARDRAILS domainIds: [################] targets: [class CheckTargetRest { type: HOST_COMMISSION resourceId: ############}] forceRefresh: true }

 

Operationsmanager log (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) shows the following entries:

yyyy-mm-ddThh:mm:ss.519+0000 DEBUG [vcf_om,################] [c.v.vcf.assessment.sdk.LambdaService,pool-3-thread-1] DAY_N_GUARDRAILS check type HOST_COMMISSION ############### status is FAIL
yyyy-mm-ddThh:mm:ss.519+0000 ERROR [vcf_om,################] [c.v.vcf.assessment.sdk.LambdaService,pool-3-thread-1] Lambda task ############### failed due to {"errorCode":"ASSESSMENT_CANNOT_ADD_CONNECTION","arguments":["nsxmgr.domain.com"],"message":"Cannot connect to nsxmgr.domain.com","cause":[{"type":"com.vmware.vapi.std.errors.Unauthorized","message":"Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {\n    messages = [],\n    data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},\n    errorType = UNAUTHORIZED\n}"}],"referenceToken":"*****"}
yyyy-mm-ddThh:mm:ss.522+0000 ERROR [vcf_om,########] [c.v.v.a.s.AssessmentLambdaServiceClient,pool-3-thread-1] Encountered exception while executing guardrail validations
java.lang.IllegalStateException: Failed to evaluate lambda class LambdaInputRest {
    checkSetType: DAY_N_GUARDRAILS
    domainIds: [######]
    targets: [class CheckTargetRest {
        type: HOST_COMMISSION
        resourceId: ######
..
..
        at java.base/java.lang.Thread.run(Thread.java:840)
yyyy-mm-ddThh:mm:ss.522+0000 ERROR [vcf_om,##########] [c.v.e.s.c.s.a.DayNGuardrailsValidationAction,pool-3-thread-1] Failed to run HOST_COMMISSION validations due to error: HOST_COMMISSION operation cannot be performed on resource ###### as these checks have failed Failed to evaluate lambda class LambdaInputRest {
    checkSetType: DAY_N_GUARDRAILS
    domainIds: [######]
    targets: [class CheckTargetRest {
        type: HOST_COMMISSION
        resourceId: ######
..
..
yyyy-mm-ddThh:mm:ss.526+0000 ERROR [vcf_om,############] [c.v.e.s.o.model.error.ErrorFactory,pool-3-thread-1] [IQNQ0E] DAYN_GUARDRAILS_VALIDATION_FAILED HOST_COMMISSION operation validation failed due to: HOST_COMMISSION operation cannot be performed on resource ###### as these checks have failed Failed to evaluate lambda class LambdaInputRest {
    checkSetType: DAY_N_GUARDRAILS
    domainIds: [######]
    targets: [class CheckTargetRest {
        type: HOST_COMMISSION
        resourceId: ######
..
..
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: HOST_COMMISSION operation validation failed due to: HOST_COMMISSION operation cannot be performed on resource ###### as these checks have failed Failed to evaluate lambda class LambdaInputRest {
    checkSetType: DAY_N_GUARDRAILS
    domainIds: [######]
    targets: [class CheckTargetRest {
        type: HOST_COMMISSION

 

Scenario 2

Decommissioning Hosts in SDDC Manager fails with below Error, 

HOST_DECOMMISSION operation validation failed due to: Failed to run guardrail validations of type HOST_DECOMMISSION on resource #######

Operationsmanager log (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) shows the following entries:

yyyy-mm-ddThh:mm:ss.830+0000 DEBUG [vcf_om,########] [c.v.vcf.assessment.sdk.LambdaService,pool-3-thread-14] DAY_N_GUARDRAILS check type HOST_DECOMMISSION ####### status is
FAIL
yyyy-mm-ddThh:mm:ss.830+0000 ERROR [vcf_om,######] [c.v.vcf.assessment.sdk.LambdaService,pool-3-thread-14] Lambda task ##### failed due to {"errorCode":"ASSESSMENT_CANNOT_ADD_CONNECTION","arguments":["nsxmgr.domain.com"],"message":"Cannot connect to nsxmgr.domain.com","cause":[{"type":"com.vmware.vapi.std.errors.Unauthorized","message":"Unauthorized

 

 

 

Environment

SDDC Manager 5.2

Cause

SDDC Manager is unable to connect to NSXT Manager.

Resolution

• Check the status of the NSXT manager credentials in SDDC Password Management and make sure they are working with the following steps and are in ACTIVE state on SDDC UI.
• View the passwords using lookup_passwords command in SDDC for NSXT_MANAGER, and confirm that these passwords are working on NSXT.
• If the NSXT manager's passwords have expired, we can perform the steps mentioned in the KB below.
• Credential operations fail on NSX-T Components in SDDC Manager