NSX Manager networking fails during VCF 9.1 upgrade due to Netplan migration error

Products

• VMware NSX

Issue/Introduction

When you upgrade VMware VCF NSX to version 9.1, you experience a failure on one or more NSX Manager nodes. You observe the following symptoms:

• The NSX Manager node loses network connectivity (no ping or SSH access).
• The vCenter console shows an error: upgrade wait_for_proton: resp_status: 500
• The ifconfig command on the affected node only shows the loopback (lo) interface; the eth0 interface is missing.
• The upgrade coordinator logs (/var/log/upgrade-coordinator/upgrade-coordinator.log) contains the following repeatedly:

Connect Async timeouts to the affected node's IP on port 9000

• As root user, in the VM console, if you run 'netplan apply', it fails with error similar to:

/etc/netplan/00-management-interface-config.yaml:6:7: Error network definition: malformed address '192.168.1.100/24/24', must be X.X.X.X/NN or X.X.X.X.X.X.X.X/NN

Environment

VMware VCF NSX 9.1

Cause

VMware VCF NSX 9.1 migrates the network configuration to Netplan. If your existing /etc/network/interfaces file contains both a CIDR (e.g., 192.168.1.100/24) and a subnet mask for the same interface, the migration script incorrectly appends both to the Netplan configuration. This results in an invalid configuration file with a double CIDR (e.g., 192.168.1.100/24/24), which causes Netplan to fail to apply the settings and results in the loss of the eth0 interface.

Resolution

To resolve this issue and restore connectivity to the failed manager:

1. Log in to the affected NSX Manager node via the vCenter console as root.
2. Open the Netplan configuration file for editing: vi /etc/netplan/00-management-interface-config.yaml
3. Locate the line containing the double CIDR notation (e.g., 192.168.1.100/24/24).
4. Remove the extra CIDR suffix so only one remains (e.g., 192.168.1.100/24).
5. Save the file and exit the editor.
6. Apply the corrected Netplan configuration: netplan apply
7. Verify that the eth0 interface is now present and has the correct IP address using ifconfig.

Preventative Steps

To prevent this issue on nodes that have not yet been upgraded:

1. Log in to the NSX Manager node as root.
2. Edit the /etc/network/interfaces file.
3. Remove any CIDR notation (e.g., /##) from the IP address entries if a subnet mask is already defined.
4. Save the file before proceeding with the upgrade to version 9.1.

Additional Information

There is a similar issue which impacts edge nodes during netplan migration NSX upgrade fails with error message 'Error migrating management interfaces to netplan.'

A similar error message can also be displayed in the NSX manager console, please review this KB NSX upgrade from 9.0.x to 9.1.0 fails with message "Upgrade has failed. Appliance OS is of a new version."