NSX Upgrade Evaluation Tool

Products

• VMware NSX

Issue/Introduction

This is a tool to detect NSX Manager data-migration related issues that can be caught before running NSX upgrades.
 
This tool is to quickly assess the readiness of the NSX Manager for upgrade. This should be used in addition to the Pre-Upgrade Checks in NSX-T when upgrading to earlier versions of 3.2.x and 4.0.x, which cover the entirety of the deployment including Edge and Host Transport Nodes.
 
There are different versions of the NSX Upgrade Evaluation Tool for use with NSX-T releases. Please ensure the version of the NSX Upgrade Evaluation Tool downloaded matches the NSX version you are upgrading to.
The NSX Upgrade Evaluation Tool download is available at the the Broadcom support portal in the My Downloads section, under the VMware NSX and VMware NSX-T Data Center sections. 
 
NSX Upgrade Evaluation Tool is now integrated with pre-upgrade checks when upgrading to 3.2.2, 4.0.1.1, and later NSX releases. Run the pre-upgrade checks as normal, and NSX will check the readiness of your NSX deployment for a successful NSX Manager upgrade. In this case, there is no need for the separate Upgrade Evaluation Tool appliance.

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Resolution

What the tool will do:

It will detect NSX Manager data-migration related issues that a customer could run into during the NSX Manager upgrade phase.

What the tool will not do:

The tool will not fix any issue that may be detected in the customer’s NSX environment.

Deliverable:

The NSX Upgrade Evaluation Tool is delivered as an independent OVA with every NSX-T Manager GA release.

System Requirements:

24GB RAM, 4 CPU. Thin Disk Provisioning is recommended for optimal use of resources. However, Thick Disk Provisioning can also be provisioned based on disk space availability.

Pre-requisites:

1. Ensure the tool can SSH into an NSX Manager node as root. This means the tool appliance will need to be able to reach any one of the existing NSX Manager nodes over TCP port 22. Also, the NSX Manager node should be able to reach the tool over TCP port 22.
2. The tool will need the user to set a password, which will follow the same guidelines of complexity as prescribed by VMware for the NSX Manager.
3. Validate that the NSX Manager cluster is healthy.  Login to the admin CLI on one of the existing NSX Manager nodes and issue the command 'get cluster status'.  All services are expected to be 'UP'. 

Recommendation:

It is recommended to run this tool twice:
1. Once before uploading the MUB and beginning the upgrade process
2. Once again before upgrading the NSX Manager nodes (after completing upgrades of all Edge and Host Transport Nodes)

Steps to run this tool:

1. Deploy the NSX Upgrade Evaluation Tool VM in your environment.
2. After deploying the VM, login to the CLI of the Upgrade Evaluation Tool VM as 'admin' user
3. Run the command: 'start dry-run data-migration mp-ip <NSX Manager Node IP>'
   a. Provide the IP of any one of the 3 NSX Manager nodes, and the root password

Tool Outcomes:

NSX Upgrade Evaluation Tool giving a go ahead for upgrade:

nsx-data-migration-dry-run> start dry-run data-migration mp-ip #.#.#.#
Root password of the Remote MP node:
Checking ssh connectivity to the MP node #.#.#.# with root user...
Checking if nub is unpacked
Untaring nub file
Unpacking nub bundle
Nub unpacked
Creating a temporary folder on MP #.#.#.# 
Copy Corfu data to the temporary folder on MP #.#.#.# 
Copy nsx_issue file to the temporary folder on MP #.#.#.# 
Create tar of the temporary folder on MP #.#.#.# 
Delete the temporary folder on MP #.#.#.# 
Fetching tar containing Corfu data
Delete tar file on MP
Downloaded corfu tgz file of size 36 MB
Loading the fetched Corfu data
Stopping Corfu server
Stopping CBM
Preparing config_bak on this VM
Untaring the data
Starting Corfu server
Starting data-migration dry-run
Running.... Please track progress in /var/log/cloudnet/data-migration.log, /var/log/proton/data-migration.log, /var/log/policy/data-migration.log, /var/log/proton/logical-migration.log

All pre-upgrade check(s) passed. You can proceed with the upgrade.

NSX Upgrade Evaluation Tool failed run example:

nsx-data-migration-dry-run> start dry-run data-migration mp-ip #.#.#.# 
Root password of the Remote MP node:
Checking ssh connectivity to the MP node #.#.#.# with root user...
Checking if nub is unpacked
Nub is already unpacked
Creating a temporary folder on MP #.#.#.#
Copy Corfu data to the temporary folder on MP #.#.#.# 
Copy nsx_issue file to the temporary folder on MP #.#.#.# 
Create tar of the temporary folder on MP #.#.#.# 
Delete the temporary folder on MP #.#.#.# 
Fetching tar containing Corfu data
Delete tar file on MP
Downloaded corfu tgz file of size 54 MB
Loading the fetched Corfu data
Stopping Corfu server
Stopping CBM
Untaring the data
Starting Corfu server
Starting data-migration dry-run
Running.... Please track progress in /var/log/cloudnet/data-migration.log, /var/log/proton/data-migration.log, /var/log/policy/data-migration.log, /var/log/proton/logical-migration.log
 
*** WARNING: Some pre-upgrade check(s) failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GSS***
LiveTraceSessionId
LiveTraceStatus

 

FAQ’s:

1. How can I test SSH connectivity from the tool to MP node?

The tool performs this as the first step when the command to run the tool is invoked. The success/failure of SSH connection is clearly reported.

2. What credentials are required to log into the CLI of the NSX Upgrade Evaluation Tool?

The credentials need to be provided by the user while deploying the NSX Upgrade Evaluation Tool ova. The complexity requirements for the password are the same as prescribed by VMware for the NSX Manager.

3. This tool requires root password for one of the 3 NSX Manager nodes. How can VMware assure me of security of my root password here?

The password is hidden on the terminal, which is similar to other NSX-T CLIs that require a password. This password is not persisted anywhere within the tool or appliance. Also, the tool is deployed within the customer’s NSX environment (within their data center) giving them complete control over data ingress and egress. Moreover, the tool does not perform any functional operations on the customer’s NSX-T environment.

4. How is root login to a Manager node enabled?

When not already enabled, to allow direct root login:

a) Open the NSX-T Manager VM console and login as root, or, SSH to the Manager as admin and elevate to root with 'st en'
b) Run 'vim /etc/ssh/sshd_config' and edit the following line:
PermitRootLogin no
to:
PermitRootLogin yes
c) Restart the ssh service:
/etc/init.d/ssh restart

This change can be reverted after using the Upgrade Evaluation Tool 

4. How does this tool ensure no impact on my NSX deployment, more specifically the NSX Manager?

The tool will create a copy of the Corfu DB on the NSX Manager appliance in the form of a TAR file and copy it into its own appliance. This TAR will then be removed from the NSX Manager ensuring complete isolation of the NSX Manager from the tool’s operations.

5. Will there be any impact on NSX Manager functionality while running this tool?

There will not be any functional impact on the NSX Manager. All other NSX Manager operations can run normally while the tool is running.

6. Why is it recommended to run this tool twice?

As NSX upgrade is often done in stages over a period of time, an end to end upgrade cycle may span across more than one typical change window, and customers may make configuration changes during these change windows. The NSX Upgrade Evaluation Tool is recommended to be run a second time right before upgrading the NSX Manager nodes (after completing upgrades of all Transport Nodes) to ensure the tool catches the impact of any changes that may have occurred during the Transport Node upgrades.

7. When can delete the deployed the NSX Upgrade Evaluation Tool?

When NSX upgrades have been completed successfully, the NSX Upgrade Evaluation Tool may be deleted from the environment.
 

8. How do we extract the support bundle from this dry run tool, considering it has no UI?

dry-run-tool> get support-bundle file dry-run-tool-support-bundle
Thu Jan 20 2022 UTC 21:02:45.782
dry-run-tool-support-bundle created, use the following command to transfer the file:
  copy file dry-run-tool-support-bundle url <url>
After transferring dry-run-tool-support-bundle, extract it using: tar xzf dry-run-tool-support-bundle

root@dry-run-tool:/image# find / -name dry-run-tool-support-bundle -print
/image/vmware/nsx/file-store/dry-run-tool-support-bundle

Use the copy admin cli command or a manual scp command to copy the bundle off the appliance and upload to VMware Support.