NSX manager reporting high memory - Upgrade Coordinator Service
search cancel

NSX manager reporting high memory - Upgrade Coordinator Service

book

Article ID: 396333

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The NSX Manager reports high memory usage.
  • The Upgrade Coordinator process is consuming high memory which under normal circumstances would typically be using less than 1GB of memory.
  • /var/log/vmware/top-mem.log in the NSX Manager reports high memory for upgrade coordinator:
       PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ TGID COMMAND

    3246815 uuc 20 0 16.0g 7.0g 9192 S 0.0 15.0 1072:05 3246815 /usr/lib/jvm/openjdk-java8-runtime-amd64/bin/java -Djava.util.logging.config.file=/opt/vmware/upgra+
  • /var/log/upgrade-coordinator/upgrade-coordinator.log in the NSX Manager node reports context initialization failed error during startup of UC service:
    2025-01-08T10:28:26.170Z WARN WrapperStartStopAppMain XmlWebApplicationContext 3319387 Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'upgradeCoordinatorStartupActions' defined in ServletContext resource [/WEB-INF/spring/upgrade-coordinator-config.xml]: Invocation of init method failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:7440/nsxapi/api/v1/uc-state": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)

    2025-01-08T10:28:26.183Z ERROR WrapperStartStopAppMain ContextLoader 3319387 Context initialization failed
  • The upgrade APIs are returning 404 in the NSX manager node post UC startup failure.
    grep ' 404 ' /var/log/upgrade-coordinator/localhost_access_log.txt | head -5
    2025-01-08T21:45:43.052Z 127.0.0.1 - - "GET /upgrade-coordinator/api/v1/upgrade/functional-state HTTP/1.1" 404 110
    2025-01-08T21:45:43.059Z 127.0.0.1 - - "GET /upgrade-coordinator/api/v1/upgrade/summary HTTP/1.1" 404 110
    2025-01-08T21:45:43.059Z 127.0.0.1 - - "GET /upgrade-coordinator/api/v1/upgrade/available-releases?source=notification HTTP/1.1" 404 110
    2025-01-08T21:48:30.212Z 127.0.0.1 - - "GET /upgrade-coordinator/api/v1/upgrade/functional-state HTTP/1.1" 404 110
    2025-01-08T21:48:30.215Z 127.0.0.1 - - "GET /upgrade-coordinator/api/v1/upgrade/summary HTTP/1.1" 404 110
  • The context initialization failed due to proton service being down at that point in time when "http://localhost:7440/nsxapi/api/v1/uc-state” API call was made.
    /var/log/proton/proton-tomcat-wrapper.log
    STATUS | wrapper | 2025/01/08 21:27:42 | TERM trapped. Shutting down.
    ERROR | wrapper | 2025/01/08 21:28:18 | Shutdown failed: Timed out waiting for signal from JVM.
    STATUS | wrapper | 2025/01/08 21:28:18 | Dumping JVM state.
    INFO | jvm 1 | 2025/01/08 21:28:18 | 2025-01-08 10:28:18

Environment

VMware NSX 

Cause

The context initialization failed due to proton service being down at that point in time when "http://localhost:7440/nsxapi/api/v1/uc-state” API call was made.

Resolution

This issue is resolved in VMware NSX 4.2.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.


Workaround: 

  • To workaround the issue, restart the upgrade coordinator service by running the below command as root user on the NSX manager nodes where the Upgrade Coordinator service is reporting high memory usage. 

    /etc/init.d/upgrade-coordinator restart 

 

Additional Information

If the workaround is not resolving the issue, collect all the 3 NSX manager log bundles and raise a support ticket with Broadcom. 

Handling Log Bundles for offline review with Broadcom support

Powered by Wolken Software