VCF Management Services API Deployment Fails with "Failed to Save or Update VCF Management Components"
search cancel

VCF Management Services API Deployment Fails with "Failed to Save or Update VCF Management Components"

book

Article ID: 451314

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • VMware Cloud Foundation (VCF) management services API deployment fails with a generic error message on the SDDC Manager UI similar to below:

    Failed to save or update VCF Management Components

  • During the failure, the domainmanager.log displays a NullPointerException during JSON Spec Validations: 

    /var/log/vmware/vcf/domainmanager/domainmanager.log

    YYYY-MM-DDThh:mm:ssDEBUG [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-43]  Invoking task AuditVcfManagementComponentsSpecActionDayN., Plugin: ValidationPlugin, ParamBuilder null, Input map: {spec=spec}, Id: [ID] ...
    YYYY-MM-DDThh:mm:ssDEBUG [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-43]  Contract task AuditVcfManagementComponentsSpecActionDayN input: {"spec":{"sddcId":"sddc_ID]","workflowType":"VCF","vcenterSpec":{"vcenterHostname":"[FQDN]","ssoDomain":"[DOMAIN]","adminUserSsoUsername":"[USER]","adminUserSsoPassword":"[PASSWORD]","useExistingDeployment":true},"dnsSpec":{"subdomain":"[DOMAIN]"},"sddcManagerSpec":{"hostname":"[FQDN]","useExistingDeployment":true},"managementPoolName":"####","vspClusterSpec":{"platformFqdn":"[FQDN]","systemUserPassword":"[PASSWORD]","ipv4Pool":{"addresses":["[IP1]","[IP2]","[IP3]","[IP4]","[IP5]","[IP6]","[IP7]","[IP8]","[IP9]","[IP10]","[IP11]","[IP12]","[IP13]",null,"[IP14]"]},"size":"[size]","internalClusterCidrIpv4":"[CIDR]","name":"[NAME]","instanceFqdn":"[FQDN]"},"fleetLcmSpec":{"size":"[size]"},"sddcLcmSpec":{"size":"[size]"},"fleetDepotSpec":{"size":"[size]"},"telemetryAcceptorSpec":{"size":"[size]"},"vidbSpec":{"hostname":"[FQDN]","size":"[size]"},"saltSpec":{"size":"[size]"},"saltRaasSpec":{"size":"[size]"},"vcfManagementComponentsInfrastructureSpec":{"localRegionNetwork":{"networkName":"[NAME]","subnetMask":"[MASK]","gateway":"[IP]"},"xRegionNetwork":{"networkName":"[NAME]","subnetMask":"[SUBNET_MASK]","gateway":"[IP]"}},"licenseServerSpec":{"hostname":"[FQDN]"},"vcfInstanceName":"[ID]"},"_executionUuid":"[ID]"}
    ...
    YYYY-MM-DDThh:mm:ssERROR [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.v.a.AuditVcfManagementComponentsSpecActionDayN,dm-exec-43]  An error occurred during JSON Spec Validations
    java.lang.NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null
    ...
    YYYY-MM-DDThh:mm:ssDEBUG [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.c.v.util.ResponseUtil,dm-exec-43]  Build validation response: {"errorCode":"JSON_SPEC_VALIDATIONS.error","arguments":[],"context":{"severity":"ERROR","bundleName":"[PACKAGE]","validation.taskId":"[ID]"},"message":"Deployment Specification Validation Failed","remediationMessage":"Verify the JSON specification is valid and contains all required fields"}
    ...
    YYYY-MM-DDThh:mm:ssERROR [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.v.s.SaveVcfManagementComponentsAction,dm-exec-71]  Custom error occurred while saving VCF Management Components
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to save or update VCF Management Components
    ...
    Caused by: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500  on PATCH request for "http://localhost/v1/system/vcf-management-components": "{"errorCode":"INVENTORY_INTERNAL_SERVER_ERROR","arguments":[],"message":"Inventory Internal Server Error","referenceToken":"[TOKEN]"}"
    ...
    YYYY-MM-DDThh:mm:ssERROR [vcf_dm, <task_id>, <subtask_id>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-71]  [TOKEN] FAILED_TO_SAVE_OR_UPDATE_VCF_MGMT_COMPONENTS Failed to save or update VCF Management Components
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to save or update VCF Management Components

  • Additionally, the commonsvcs.log displays a database constraint violation due to the unsupported null entry derived from the malformed API payload:

    /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

    YYYY-MM-DDThh:mm:ss INFO [common,<task_id>, <subtask_id>] [c.v.e.s.i.r.a.c.v1.SystemController,http-nio-127.0.0.1-7100-exec-9] Saving VCF Management Components rest model: {"vspCluster":{"fqdn":"[FQDN]","deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"sddcLcm":{"fqdn":"[FQDN]","deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"fleetLcm":{"deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"telemetryAcceptor":{"deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"vidb":{"fqdn":"[FQDN]","deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"salt":{"deploymentStatus":"NOT_STARTED","deploymentType":"NEW"},"saltRaas":{"deploymentStatus":"NOT_STARTED","deploymentType":"NEW"}} 
    YYYY-MM-DDThh:mm:ss INFO [common,<task_id>, <subtask_id>] [c.v.e.s.i.r.a.c.v1.SystemController,http-nio-127.0.0.1-7100-exec-9] VCF Management Components after translation to inventory model: [VcfManagementComponent(componentType=VSP, fqdn=[FQDN], deploymentStatus=NOT_STARTED, deploymentType=NEW), VcfManagementComponent(componentType=SDDC_LCM, fqdn=[FQDN], deploymentStatus=NOT_STARTED, deploymentType=NEW), VcfManagementComponent(componentType=FLEET_LCM, fqdn=null, deploymentStatus=NOT_STARTED, deploymentType=NEW)] 
    YYYY-MM-DDThh:mm:ss WARN [common,<task_id>, <subtask_id>][o.h.e.jdbc.spi.SqlExceptionHelper,http-nio-127.0.0.1-7100-exec-9] SQL Error: 0, SQLState: #### 
    YYYY-MM-DDThh:mm:ss ERROR [common,<task_id>, <subtask_id>][o.h.e.jdbc.spi.SqlExceptionHelper,http-nio-127.0.0.1-7100-exec-9] ERROR: null value in column "fqdn" of relation "vcf_management_component" violates not-null constraint Detail: Failing row contains (FLEET_LCM, null, NEW, NOT_STARTED). 
    YYYY-MM-DDThh:mm:ss ERROR [common,<task_id>, <subtask_id>] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-9] [[TOKEN]] INVENTORY_INTERNAL_SERVER_ERROR Inventory Internal Server Error com.vmware.evo.sddc.inventory.model.InventoryManagerException: Inventory Internal Server Error at com.vmware.evo.sddc.inventory.dal.sql.client.VcfManagementComponentsClientImpl.createOrUpdateVcfManagementComponents(VcfManagementComponentsClientImpl.java:77)

Environment

VCF 9.1

Cause

The cause is a syntax error in the API JSON request, in this specific scenario an extra comma within the ipv4Pool.addresses array between "[IP13]" and "[IP14]" which results in a null value being parsed into the inventory model as evident on the domainmanager.log. While the AuditVcfManagementComponentsSpecActionDayN validation flags the error, the orchestration may partially proceed, attempting to save the component to the database. Consequently the FLEET_LCM component entry fails database insertion because the fqdn column enforces a strict not-null constraint which is violated by the null value derived from the malformed JSON as evident from the vcf-commonsvcs.log

 

Resolution

  1. Take a virtual machine snapshot of the SDDC Manager appliance to safely prepare for database cleanup.

  2. Review and remove any errant commas or structural defects generating null entries within the API payload using jq command as per Cluster Creation Validation Fails with JSON Syntax Errors in SDDC Manager

    jq empty <API_spec>.json

  3. Follow KB 445490: VCF Management Component Deployment fails with "existing deployment" error to remove orphan entries from failed attempt in the SDDC DB and restart the SDDC Manager services

  4. Validate the deployment JSON payload using the Validation API on SDDC Manager to ensure there are no errors flagged and proceed to resubmit the JSON payload to trigger the VCF Management components installation workflow by following Deploy VCF Management Services via API

Additional Information

 

 

 

Powered by Wolken Software