Adding a new host fails when scaling out a vSAN cluster that has NSX and vLCM enabled

Products

• VMware NSX
• VMware vCenter Server
• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

• Scaling out a vSAN cluster with NSX C-VDS and vLCM enabled by adding a new host directly to the cluster fails due to the errors below:
• FileService (FS) may or may not be in use.
• You may see an error from EAM/FS that FS agent VM cannot deploy when adding host to cluster.
• You get a failing task of "Apply NSX Solution" with an error similar to:

A general system error occur red: Health Check for 'Cluster *' failed

• In  NSX Manager, you get a failed node with an error similar to:

Failed to install software on host. Solution apply failed on host: '#.#.#.#' vSAN health test 'vSAN cluster partition' reported an issue. Check the vSAN health.vSAN health test 'All hosts have a vSAN vmknic configured' reported an issue. Check the vSAN health.Solution apply failed on host: 'x.x.x.x
​​​​

Note: It is expected that the NSX installation completes automatically

Environment

VMware vSphere ESXi
VMware vCenter Server
VMware NSX
VMware vSAN

Cause

The error occurs due to the vSAN Health check failures related to vmknic connections. You see errors similar to the following on the Monitor tab for the cluster in the vSphere Client, under the Skyline Health section:

All hosts have a VSAN vmknic configured

NSX expects a healthy vSAN enabled vmknic to be configured when adding a new node.

Resolution

This is a condition that may occur in a VMware NSX environment.

 

Workaround:

1. Before scaling out the vSAN cluster with NSXT-CVDS and vLCM enabled with a new host, verify there are no cluster health check errors
2. Add a new host in the Datacenter and put it in maintenance mode
3. Add the new host to the VDS
4. Install and config NSX on the new host
5. Create the vSAN vmknic on the host
6. Exit the host from maintenance mode
7. Add the host into the cluster
8. Create disk group on the new host (EAM will retry the FS agent deployment), if required.

Note:
In step 7:

• The EAM will try to deploy FileService agent, but as no disk group is created on the new host yet, the task will fail and prompt error alert.
• The "Apply NSX Solution"(with vLCM remediation) task will still be triggered and fail due to the EAM error alert, but the NSX component is installed/configured in step 4 already

In step 8:

• After disk group is created, wait for a while, "Remediate vSAN file service" will be triggered and the EAM will retry the FileService agent deployment successfully. The cluster with the new host goes back to healthy status and the VMs starts to migrate to this new host.