NSX unable to upgrade due to java.lang.NullPointerException
search cancel

NSX unable to upgrade due to java.lang.NullPointerException

book

Article ID: 370877

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • In VMware NSX 3.2.3 or any version above it, while attempting to upgrade to a later version, the NSX may show throw the error "java.lang.NullPointerException" without any other explanation of the issue. The repo-sync status may also fail for some of the NSX manager nodes. 
  • This usually occurs after the pre-check is completed and proceeds to upgrade the NSX edges (as the first part of the upgrade). 
  • In the /var/log/upgrade-coordinator/upgrade-coordinator.log, the following entries are seen:
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 UpgradeCoordinatorFacadeImpl 1541063 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component: EDGE, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 UpgradeCoordinatorFacadeImpl 1541063 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component: HOST, status: SUCCESS, % complete: 100.0, details: null, canSkip: false
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 UpgradeCoordinatorFacadeImpl 1541063 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component: MP, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] In setOverallUpgradeStatus for PAUSED, upgradeExecutionContext.getUpgradeStatus(): PAUSED
YYYY-MM-DDTHH:MM:SS.###Z ERROR http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component EDGE
java.lang.NullPointerException: null
        at com.google.protobuf.StringValue$Builder.setValue(StringValue.java:472) ~[protobuf-java-3.21.7.jar:?]
        at com.vmware.nsx.management.upgrade.persistence.service.impl.UpgradeHistoryPersistenceServiceImpl.setUpgradeHistory(UpgradeHistoryPersistenceServiceImpl.java:102) ~[libuc-nsx-commons.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.product.NsxUpgradePlugin.setUpgradeHistory(NsxUpgradePlugin.java:1090) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.pluginframework.PluginManager.setUpgradeHistory(PluginManager.java:401) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.updateUpgradeHistory(UpgradeServiceImpl.java:1046) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.continueUpgrade(UpgradeServiceImpl.java:282) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl.continueUpgrade_aroundBody52(UpgradeCoordinatorFacadeImpl.java:884) ~[libuc-

YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 FacadeInterceptorHelperImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Caught error in facade interceptor
java.lang.NullPointerException: null
        at com.google.protobuf.StringValue$Builder.setValue(StringValue.java:472) ~[protobuf-java-3.21.7.jar:?]
        at com.vmware.nsx.management.upgrade.persistence.service.impl.UpgradeHistoryPersistenceServiceImpl.setUpgradeHistory(UpgradeHistoryPersistenceServiceImpl.java:102) ~[libuc-nsx-commons.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.product.NsxUpgradePlugin.setUpgradeHistory(NsxUpgradePlugin.java:1090) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.pluginframework.PluginManager.setUpgradeHistory(PluginManager.java:401) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.updateUpgradeHistory(UpgradeServiceImpl.java:1046) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.continueUpgrade(UpgradeServiceImpl.java:282) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl.continueUpgrade_aroundBody52(UpgradeCoordinatorFacadeImpl.java:884) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.upgrade.facade.UpgradeCoordinatorFacadeImpl$AjcClosure53.run(UpgradeCoordinatorFacadeImpl.java:1) ~[libuc-core.jar:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.9.1.jar:1.9.1]
        at com.vmware.nsx.management.upgrade.facade.FacadeInterceptorHelperImpl.proceed(FacadeInterceptorHelperImpl.java:76) ~[libuc-core.jar:?]
        at com.vmware.nsx.management.common.api.FacadeInterceptorAspect.annotatedMethod(FacadeInterceptorAspect.java:80) ~[libmp_common.jar:?]

YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 AuditingServiceImpl 745049 - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] UserName="admin", Src="example.com", ModuleName="Upgrade", Operation="ContinueUpgrade", Operation status="failure", New value=[{"skip":true,"component_type":"EDGE"}]
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-2 FacadeInterceptorHelperImpl 745049 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Caught error in facade interceptor
java.lang.NullPointerException: null
        at com.google.protobuf.StringValue$Builder.setValue(StringValue.java:472) ~[protobuf-java-3.21.7.jar:?]
        at com.vmware.nsx.management.upgrade.persistence.service.impl.UpgradeHistoryPersistenceServiceImpl.setUpgradeHistory(UpgradeHistoryPersistenceServiceImpl.java:102) ~[libuc-nsx-commons.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.product.NsxUpgradePlugin.setUpgradeHistory(NsxUpgradePlugin.java:1090) ~[libnsx-upgrade-plugins.jar:?]
  • The "source_version" and/or "target_version" is set to null.
YYYY-MM-DDTHH:MM:SS.###Z  INFO localhost-startStop-1 UpgradePlanServiceImpl 966306 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Persisting Plan {"upgrade_status":"NOT_STARTED","source_version":null,"target_version":null,"component_plans":{"MP":{"component_type":
YYYY-MM-DDTHH:MM:SS.###Z  WARN NSXUcUpgradeService-3-1 NSXUcUpgradeStep 3934146 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] The version null is not recognized by the Manager. Please ensure that Repository is in sync with the other nodes.
com.vmware.nsx.management.upgrade.exceptions.UcMethodExecutionFailedException: com.vmware.nsx.management.upgrade.rpcframework.UcRestRpcException: [UC] Error in rest call. url= /api/v1/cluster/<node-UUID>/node/upgrade/uc , method= POST , response= {"error_code": 37116, "error_message": "The version null is not recognized by the Manager. Please ensure that Repository is in sync with the other nodes.", "module_name": "node-services"} , error= 400 Bad Request: "{"error_code": 37116, "error_message": "The version null is not recognized by the Manager. Please ensure that Repository is in sync with the other nodes.", "module_name": "node-services"}" .
YYYY-MM-DDTHH:MM:SS.###Z  INFO http-nio-127.0.0.1-7442-exec-4 UpgradeCoordinatorFacadeImpl 3183 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Got upgrade summary systemVersion: 4.1.2.1.0.22667789, status: PAUSED, targetVersion: null, ucVersion: 4.1.2.3.0.23382420, upgradeBundleFileName: VMware-NSX-upgrade-bundle-4.1.2.3.0.23382408-pre-check.pub
  • The "source_version" and/or the "target_version" is incorrectly set to a value that does not match the actual version. For example, the actual source version is 4.2.1.4.0.24765227, but it is currently listed as 3.1.3.5.0.19068434.

    YYYY-MM-DDTHH:MM:SS.###Z  INFO WrapperStartStopAppMain UpgradePlanServiceImpl 3427187 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Persisting Plan {"upgrade_status":"NOT_STARTED","source_version":"3.1.3.5.0.19068434","target_version":"4.2.3.1.0.24954727","component_plans":{"MP":{"component_type":"MP",

Environment

VMware NSX 3.2.3 and above

VMware NSX

Cause

This issue may seem to be related to the NSX Edges, however it is not. The NSX Upgrade plan was reset and the target version was set to "null". This could be triggered by restarting the install-upgrade process on the NSX Managers. 

Resolution

This issue is fixed in future releases.

Workaround:

  1. Run following command on all the three nodes: #: /etc/init.d/upgrade-coordinator stop
  2. Send a GET API call to https://<manager>/api/v1/upgrade-mgmt/plan and save the JSON output to a text file for the next step.
    Make sure to add these 2 header when you push the API: (same from CLI or Postman): "Content-Type: text/plain" , "Accept: text/plain".
    # curl -X  GET -H 'Content-Type: text/plain' -H "Accept: text/plain" -k -u admin  "https://nsx-mngr-01.corp.local/api/v1/upgrade-mgmt/plan"
  3. Change ("source_version":null,"target_version":null) to ("source_version":"<current-source-version>","target_version":"<target-upgrade version>")
    • An example:  ("source_version":"3.2.3.0.0.21703624","target_version":"4.1.2.3.0.23382408")
  4. Send a PUT API call to https://<manager>/api/v1/upgrade-mgmt/plan with the entire modified text from above GET API call inside the body of the PUT request
  5. Run the following on all the three nodes: #: /etc/init.d/upgrade-coordinator start.
Powered by Wolken Software