VCF Management domain deployment using Cloud builder fails at Configure Customer Experience Improvement Program (CEIP) on NSX Data Center Managers
search cancel

VCF Management domain deployment using Cloud builder fails at Configure Customer Experience Improvement Program (CEIP) on NSX Data Center Managers

book

Article ID: 382996

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware Cloud Foundation 5.x VMware SDDC Manager

Issue/Introduction

  • VCF Management domain deployment using Cloud builder fails at Configure Customer Experience Improvement Program (CEIP) on NSX Data Center Managers
  • Error in Cloud builder UI 
    Failed to configure Customer Experience Improvement Program (CEIP) on NSX Data Center Manager <NSXT_MANAGER_FQDN>
  • NSXT Manager UI has the below error 
    Error: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111 (Error code: 98)
  • Error in vcf-bringup-debug.log 
    2024-11-26T05:06:28.980+0000 [bringup,6745575439dd683b9ead1d2c0f36c633,4a84] ERROR [c.v.v.c.n.s.c.c.ComplexHelpers,pool-2-thread-14] Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [],
    data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
    errorType = INTERNAL_SERVER_ERROR
}

Caused by: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [],
    data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
    errorType = INTERNAL_SERVER_ERROR
}
    2024-11-26T05:06:28.981+0000 [bringup,6745575439dd683b9ead1d2c0f36c633,4a84] ERROR [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-14] [UMFRHG] FAILED_TO_CONFIGURE_CEIP Failed to configure Customer Experience Improvement Program (CEIP) on NSX  Data Center Manager <NSXT_MANAGER_FQDN>.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to configure Customer Experience Improvement Program (CEIP) on NSX  Data Center Manager <NSXT_MANAGER_FQDN>.

Caused by: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [],
    data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
    errorType = INTERNAL_SERVER_ERROR
  • Configuring CEIP manually in NSXT Manager (NSXT Manager UI > System > General Settings > Customer Program) fails with below error 
    # Error: Uncaught TypeError: Cannot read properties of undefined (reading 'items')
# File: https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011
# Line | Column:5 | 8614
# StackTrace: TypeError: Cannot read properties of undefined (reading &#39;items&#39;)
at populate0bj (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:5:8614)
at constructor.updateRecord (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:5:11491)
at constructor.processSubmitData (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:10:16416)
at constructor.callParent (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:19:3586)
at constructor.processSubmitData (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:131:25376)
at constructor.onClickSubmit (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:10:13246)
at Object.callback (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:17:29213)
at constructor.fireHandler (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:59:11608)
at constructor.onClick (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:59:11387)
at constructor.fire (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:21:22392)
at constructor.fire (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26604)
at constructor.publish (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26237)
at constructor.doDelegatedEvent (https://<NSXT_MANAGER_FQDN>//nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26968)
at constructor.onDelegatedEvent (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26740)
at method (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:17:13185)|
  • Error in NSXT Manager /var/log/phonehome-coordinator/phonehome-coordinator-tomcat-wrapper.log 
    INFO   | jvm 1    | 2024/11/26 04:50:07 | Terminating due to java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper  | 2024/11/26 04:50:07 | The JVM has run out of memory.  Requesting thread dump.
STATUS | wrapper  | 2024/11/26 04:50:07 | Dumping JVM state.
ERROR  | wrapper  | 2024/11/26 04:50:07 | Could not dump JVM state: No such process
STATUS | wrapper  | 2024/11/26 04:50:12 | JVM process is gone.

Environment

VMware Cloud Foundation 5.x

Cause

The cause is due to a race condition. The phonehome-coordinator service crashes and won't start because of out of a memory issue during initializing time.

Resolution

Powered by Wolken Software