Replication errors during NSX upgrade in Cross vCenter environment

Products

• VMware NSX Networking

Issue/Introduction

This article is to highlight the documentation in regards to the sync issues displayed.

Symptoms:
After upgrading the Primary NSX Manager, and before upgrading the Secondary NSX Manager(s):

• You see the following sync issues in vSphere Web Client under Networking & Security / Installation and Upgrade:

Requested URL /api/2.0/services/application/dpi/discovered-appguid could not be found. This may be caused by mismatch in NSX manager versions or missing object ID.

To further identify this condition, the following matches can be found in the NSX Manager logs:

• In localhost_acess_log_date.txt:

127.0.0.1 - - [07/Jun/2018:09:58:27 +0200] "POST /api/2.0/services/application/dpi/discovered-appguid HTTP/1.1" 404 1018 16

• In replicator.log:

​2018-06-07 09:58:30.578 CEST ERROR pool-4-thread-1 ReplicatorUtils:84 - - [nsxv@6876 comp="nsx-manager" errorCode="MP160307" subcomp="manager"] Error in calling POST for object <?xml version="1.0" encoding="UTF-8"?>
<appguid><appGuidId>1697</appGuidId><appGuidName>GITHUB</appGuidName></appguid> on VSM 10.140.0.220
org.springframework.web.client.HttpClientErrorException: 404 Not Found
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:85) ~[spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:708) ~[spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:661) ~[spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621) ~[spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
        at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:388) ~[spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
        at com.vmware.vshield.replicator.utils.VsmRestClient.post(VsmRestClient.java:148) ~[classes/:?]
        at com.vmware.vshield.replicator.utils.ReplicatorUtils.executeRestApi(ReplicatorUtils.java:53) [classes/:?]
        at com.vmware.vshield.replicator.providers.DefaultReplicationProvider.executeRestApi(DefaultReplicationProvider.java:302) [classes/:?]
        at com.vmware.vshield.replicator.providers.DefaultReplicationProvider.handleCreate(DefaultReplicationProvider.java:74) [classes/:?]
        at com.vmware.vshield.replicator.providers.DefaultReplicationProvider.handleCreateInternal(DefaultReplicationProvider.java:348) [classes/:?]
        at com.vmware.vshield.replicator.providers.DefaultReplicationProvider.handleFullSync(DefaultReplicationProvider.java:187) [classes/:?]
        at com.vmware.vshield.replicator.queuemgmt.SecondaryReplicationQueue$SecondaryEventDispatcher.run(SecondaryReplicationQueue.java:156) [classes/:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x

Cause

The sync issues occur because the REST API call used for the specific object exists in the upgraded Primary NSX Manager but not in the non-upgraded Secondary NSX Manager(s).

Resolution

As documented in NSX 6.4 Upgrade Guide:

If you are logged in to the vSphere Web Client during the upgrade, you will see synchronization issue warnings on the Networking and Security > Installation and Upgrade > Management page. This is because you have NSX Manager appliances with different versions of NSX. You must upgrade the secondary NSX Manager appliances before proceeding with any other part of the upgrade.

Proceed with the upgrade of the Secondary NSX Manager(s). The sync issues will be automaticall cleared after all Secondary NSX Managers are upgraded.

Additional Information

Impact/Risks:
This does not cause any impact.