Failed to configure Customer Experience Improvement Program (CEIP) on NSX Data Center Manager <NSXT_MANAGER_FQDN>
Error: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111 (Error code: 98)
2024-11-26T05:06:28.980+0000 [bringup,6745575439dd683b9ead1d2c0f36c633,4a84] ERROR [c.v.v.c.n.s.c.c.ComplexHelpers,pool-2-thread-14] Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
messages = [],
data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
errorType = INTERNAL_SERVER_ERROR
}
Caused by: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
messages = [],
data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
errorType = INTERNAL_SERVER_ERROR
}
2024-11-26T05:06:28.981+0000 [bringup,6745575439dd683b9ead1d2c0f36c633,4a84] ERROR [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-14] [UMFRHG] FAILED_TO_CONFIGURE_CEIP Failed to configure Customer Experience Improvement Program (CEIP) on NSX Data Center Manager <NSXT_MANAGER_FQDN>.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to configure Customer Experience Improvement Program (CEIP) on NSX Data Center Manager <NSXT_MANAGER_FQDN>.
Caused by: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
messages = [],
data = struct => {error_message=upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111, error_code=98, module_name=common-service},
errorType = INTERNAL_SERVER_ERROR
# Error: Uncaught TypeError: Cannot read properties of undefined (reading 'items')
# File: https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011
# Line | Column:5 | 8614
# StackTrace: TypeError: Cannot read properties of undefined (reading 'items')
at populate0bj (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:5:8614)
at constructor.updateRecord (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:5:11491)
at constructor.processSubmitData (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:10:16416)
at constructor.callParent (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:19:3586)
at constructor.processSubmitData (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:131:25376)
at constructor.onClickSubmit (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-all.min.js?rel=1726773080011:10:13246)
at Object.callback (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:17:29213)
at constructor.fireHandler (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:59:11608)
at constructor.onClick (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:59:11387)
at constructor.fire (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:21:22392)
at constructor.fire (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26604)
at constructor.publish (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26237)
at constructor.doDelegatedEvent (https://<NSXT_MANAGER_FQDN>//nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26968)
at constructor.onDelegatedEvent (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:25:26740)
at method (https://<NSXT_MANAGER_FQDN>/nsxmanager/nsx-libs.min.js?rel=1726773080011:17:13185)|
INFO | jvm 1 | 2024/11/26 04:50:07 | Terminating due to java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | 2024/11/26 04:50:07 | The JVM has run out of memory. Requesting thread dump.
STATUS | wrapper | 2024/11/26 04:50:07 | Dumping JVM state.
ERROR | wrapper | 2024/11/26 04:50:07 | Could not dump JVM state: No such process
STATUS | wrapper | 2024/11/26 04:50:12 | JVM process is gone.
VMware Cloud Foundation 5.x
The cause is due to a race condition. The phonehome-coordinator service crashes and won't start because of out of a memory issue during initializing time.