VVF deployment fails while populating SDDC Manager inventory and credentials with an error "Invalid HTTP status code [0] while configuring SDDC Manager Inventory"
search cancel

VVF deployment fails while populating SDDC Manager inventory and credentials with an error "Invalid HTTP status code [0] while configuring SDDC Manager Inventory"

book

Article ID: 408583

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This article provides steps to identify the cause for VCF 9.0 deployment failure while populating SDDC manager inventory and credentials for ESX hosts. Task progress shows failed status with below error message ;

  • Issue is caused to stale entries of ESX host found in database due to past deployment failures. Trace the error message/s as shown below;
  • /var/log/vmware/vcf/domainmanager/domainmanager.log --- error is shown while configuring SDDC manager inventory

YYYY-MM-DDTHH:MIN DEBUG [vcf_dm,####,7f8f] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-33]  Security config retrieved {"fipsMode":true}
YYYY-MM-DDTHH:MIN ERROR [vcf_dm,####,7f8f] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-33]  [token id] VCF_GUEST_SERVICE_INVALID_HTTP_STATUS_CODE Invalid HTTP status code {0}, while configuring SDDC Manager inventory
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Invalid HTTP status code {0}, while configuring SDDC Manager inventory
        at com.vmware.evo.sddc.sddcmanager.PostInventoryAndCredentialsAction.execute(PostInventoryAndCredentialsAction.java:124)
        at com.vmware.evo.sddc.sddcmanager.PostInventoryAndCredentialsAction.execute(PostInventoryAndCredentialsAction.java:47)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:517)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
        at jdk.internal.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : "{"errorCode":"INVENTORY_INTERNAL_SERVER_ERROR","arguments":[],"message":"Inventory Internal Server Error","referenceToken":"token id"}"
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:102)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:672)
        at com.vmware.evo.sddc.sddcmanager.PostInventoryAndCredentialsAction.execute(PostInventoryAndCredentialsAction.java:73)
        ... 16 common frames omitted
YYYY-MM-DDTHH:MIN DEBUG [vcf_dm,68adb6e284fe607a81f6a6c78316dba8,7f8f] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-33]  Collected the following errors for task with name PostInventoryAndCredentialsAction and ID ####-####: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.sddcmanager.messages)]]

  • /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log list the ESX FQDN as already exist

YYYY-MM-DDTHH:MIN_SEC INFO  [common,####,554d] [c.v.e.s.i.s.EsxiInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-2] Create Esxi - ####
YYYY-MM-DDTHH:MIN ERROR [common,####,554d] [c.v.e.s.i.s.EsxiInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-2] ESXi <ESX FQDN> already exist: ##### <ESX FQDN> already exists
YYYY-MM-DDTHH:MIN ERROR [common,####,554d] [c.v.e.s.i.s.e.DomainExtensionsImpl,http-nio-127.0.0.1-7100-exec-2] Failed to post limited domain inventory com.vmware.evo.sddc.inventory.model.InventoryAlreadyExistsException: ####-####-#### <ESX FQDN> already exists

Environment

VCF 9.0

VVF 9.0

Resolution

This is a known issue and shall be fixed in upcoming VCF release.

Workaround steps:

Remove stale entries of host from database. Take a backup of VCF Installer/SDDC before proceeding with below mentioned steps.

  • SSH as VCF user to VCF Installer via Putty client 
  • Switch to root using 'su' command
  • Enter postgres database use the command: psql -h localhost -U postgres -d platform 
  • List host names from platform table. Output will be similar to;

platform=# select hostname from host where status!='ACTIVE'; --- shows the list of hosts that aren't in active state.

hostname
------------
 esx01.fqdn
 esx02.fqdn
(2 rows)

  • Delete host name/s from the table;

platform=# delete from host where hostname='esx fqdn from output';
DELETE 1

  • Restart the deployment task.

You will read successfully completion of VVF deployment in /var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDTHH:MIN_SEC DEBUG [vcf_dm,####,ba32] [c.v.e.s.c.s.v.impl.VcfOpsServiceImpl,dm-exec-24]  Received VCF Operations deployment cluster info: {"online_state":"GOING_ONLINE","initialization_state":"INITIALIZING","installation_state":"CONFIGURED"}
YYYY-MM-DDTHH:MIN_SEC DEBUG [vcf_dm,68af37bf5ac4cf5acd9a550dd293f036,ba32] [c.v.e.sddc.vrealize.util.PollingUtil,dm-exec-24]  Polling status: {"online_state":"GOING_ONLINE","initialization_state":"INITIALIZING","installation_state":"CONFIGURED"}

Powered by Wolken Software