"Unable to list SCSI HBAs. API cannot be invoked with EXTERNAL context during boot" error after upgrading ESXi host to 7.0 U1 bootbank, altbootbank is not listed.

Products

• VMware vSphere ESXi

Issue/Introduction

• Post ESXi host upgrade to 7.0U1, the bootbank and altbootbank directories are not visible as shown in the below screenshot
• Similar log entries in ESXi host logs /var/log/jumpstart-native-stdout.log

2020-11-16T19:49:52.695Z| Method start executed successfully for plugin vsan-devel
This script is deprecated
2020-11-16T19:49:52.728Z| Method start executed successfully for plugin nvme-nguid-config
[ConfigStore:843020364928:] [2000] API cannot be invoked with EXTERNAL context during boot
FcoeConfig: Unable to write config for nic:vmnic0 e:0 priority:18446744073709551615 vn2vn:0-65535 Reason:API cannot be invoked with EXTERNAL context during boot
Traceback (most recent call last):
  File "/build/mts/release/bora-16850804/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/systemStorage/esxdisk.py", line 65, in execLocalcli
  File "/build/mts/release/bora-16850804/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/esxutils.py", line 220, in runCli
esxutils.EsxcliError: Errors:
Unable to list SCSI HBAs. API cannot be invoked with EXTERNAL context during boot

• NOTE :- There is a similar symptom , where post upgrade to vSphere 7.0 Update 1 the Bootbank loads to /tmp/ and The storage-path-claim service claims the storage device ESXi is installed on during startup. This causes the bootbank/altbootbank image to become unavailable and the ESXi host reverts to ramdisk.

Please refer the KB for this issue :- Bootbank loads in /tmp/ after reboot of ESXi 7.0 Update 1 host

Environment

VMware vSphere ESXi 7.0.0

Cause

• During boot, ESXi attempts to update the Software FCOE configuration for a nic and the resulting configstore write failures eventually leads to incorrect/missing bootbank/altbootbank symlinks.
• From vSphere ESXi 7.0 onwards Software FCoE is not supported in production environments. Please refer this link for more details.

 

Resolution

This issue is resolved in vSphere ESXi 7.0 P02/U1c, available at VMware vSphere.

Workaround:

1. To view all Software FCoE configurations use the below command
   • configstorecli config current get -c esx -g storage_fcoe -k fcoe_activation_nic_policies
2. Run the below steps on the ESXi 7.0 hosts where the bootbank and altbootbank symlinks are broken, Do not run this on the ESXi host where the symlinks are intact.

Make sure Software FCOE is not used before removing these configurations.

Run below command to remove all Software FCoE configuration from configstore.

• configstorecli config current delete -c esx -g storage_fcoe -k fcoe_activation_nic_policies --all 
• Or identify individual nics and try to delete these configuration using below command,
• configstorecli config current delete -c esx -g storage_fcoe -k fcoe_activation_nic_policies  -i <vmnic-#>
  • Example:

[[email protected]:~] configstorecli config current get -c esx -g storage_fcoe -k fcoe_activation_nic_policies
[
   {
      "enabled": false,
      "nic": "vmnic0",
      "vn2vn_mode": false
   },
   {
      "enabled": false,
      "nic": "vmnic1",
      "vn2vn_mode": false
   }
]
[[email protected]:~] configstorecli config current delete -c esx -g storage_fcoe -k fcoe_activation_nic_policies --all
Delete: completed successfully
[[email protected]:~] configstorecli config current get -c esx -g storage_fcoe -k fcoe_activation_nic_policies
{}

3. Reboot the ESXi host for the changes to take effect.
4. Re-try the ESXi host upgrade to 7.0U1.