"Failed to remove NSX ownership due to error Error in rest call" error and Compute Manager status down

Products

• VMware NSX

Issue/Introduction

• On System -> Fabric -> Compute Manager is Down with error "Failed to remove NSX ownership due to error Error in rest call"

• Scrolling down on the UI error above, it includes the text
  
  https://sdkTunnel:8089/sdk/vimService invocation failed with \\\"java.net.SocketException: Connection reset
  
  
• Selecting the Error and clicking Resolve does not change the status
• The issue is possibly seen after an environmental change, vCenter certificate change, network outage, appliance restart etc.
• NSX Manager log /var/log/cm-inventory/cm-inventory.log has entries similar to this example
  
  <DATE>T<TIME>Z  INFO InventoryFetcher-<id> RetryExec 5086 I/O exception (java.net.SocketException) caught when processing request to {tls}->http://vcenter.example.com:80->https://sdkTunnel:8089: Connection reset
  
  
• Required ports from NSX Manager to vCenter 80 and 443 are open from all 3 Managers to 
  
  # nc -v -w3 vc.example.com 443
Connection to vc.example.com 443 port [tcp/https] succeeded!

# nc -v -w3 vc.example.com 80
Connection to vc.example.com 80 port [tcp/http] succeeded!
  
  
• A tunnelled ssl connection over port 80 shows the same Connection reset error as seen on the UI
  
  As root user on an NSX Manager:
  # curl -k -v -p -x http://vcenter.example.com:80 https://sdkTunnel:8089
*   Trying 192.0.2.1:80...
* Connected to (nil) (192.0.2.1) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to sdkTunnel:8089
> CONNECT sdkTunnel:8089 HTTP/1.1
> Host: sdkTunnel:8089
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< content-length: 0
* Ignoring Content-Length in CONNECT 200 response
< server: envoy
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to sdkTunnel:8089
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to sdkTunnel:8089
  
  

Environment

VMware NSX

Cause

NSX Manager uses an ssl tunnel over port 80 to vCenter. Some physical firewalls will block ssl traffic over port 80 which will result in Compute Manager Down.

Resolution

Contact the security team responsible for physical firewall management and request to allow ssl traffic over port 80 between NSX Manager and vCenter.