iSCSI target configuration is lost following ESXi reboot, on 7.0 host which had been upgraded from ESXi 6.7

Products

• VMware vSphere ESXi

Issue/Introduction

On ESXi hosts which have been upgraded from ESXI 6.7 to 7.0, the hosts lose the iSCSI target configuration after reboot. The IQN remains the same and matches with the initiator group on the storage, but all configured targets, static or dynamic, are gone.

The configuration can be reconfigured, and access to the iSCSI datastores restored. However, the issue recurs on reboot.

 

vmkernel logs report multiple plugins fail to start on ESXi boot, e.g.:

Jumpstart plugin fcoe-config activation failed: fcoe-config->start() failed: error while executing the cli
Jumpstart plugin auto-configure-networking activation failed: auto-configure-networking->start() failed: error while executing the cli
Jumpstart plugin vmotion activation failed: vmotion->start() failed: error while executing the cli
Jumpstart plugin nfs41 activation failed: nfs41->start() failed: error while executing the cli
Jumpstart plugin nfs activation failed: nfs->start() failed: error while executing the cli
Jumpstart plugin software-iscsi activation failed: software-iscsi->start() failed: error while executing the cli

Environment

VMware vSphere ESXi 7.0.x

Cause

This issue arises if the host, when on 6.7 had a FCoE software adapter configured and this was not removed prior to upgrade. The ESXi 7.0 host retains the FCoE software adapter,  However, the software FCoE adapter is not supported in 7.0.

This misconfiguration causes the FCoE plugin to fail to load and may cause the iSCSI adapter to fail to load, leading to the observed symptoms.

Resolution

To resolve this issue, follow these steps to disable Software FCoE:

1. Open an SSH session to the ESXi host:

2. Run the following command to check for any existing Software FCoE configurations:
   
   configstorecli config current get -c esx -g storage_fcoe -k fcoe_activation_nic_policies
   

3. If any configurations are found, remove them using one of the following commands:
   
   a. To remove Software FCoE on all NICs:

      configstorecli config current delete -c esx -g storage_fcoe -k fcoe_activation_nic_policies --all

   b. To remove Software FCoE on a specific NIC (replace <vmnic-#> with the actual NIC name):

      configstorecli config current delete -c esx -g storage_fcoe -k fcoe_activation_nic_policies -i <vmnic-#>

 

4. Verify that the Software FCoE configuration has been removed by running the command from step 2 again. The output should be empty ({}).

 

5. Reboot the ESXi host.

 

6.Confirm that the iSCSI target configuration is present and that the iSCSI datastores are accessible. 

Additional Information

The same underlying issue may also cause vMotion failures. See vMotion Fails with "Invalid Fault" Error on ESXi Host updated to 7.0 or newer  

On update to ESXi 8.x, any software FCoE configuration created under ESXi 6.7 is not persisted.