Virtual machine replication fails with error NFC_SESSION_ERROR

Products

• VMware vSphere ESXi

Issue/Introduction

• VM replication fails with NFC_SESSION_ERROR.
• The errors are seen on replication appliances provided by Commvault, VMware vSphere Replication, etc.
• Below are the error snippets found in /var/run/log/hostd.log:

YYYY-MM-DDTHH:MM:SSZ info hostd[2103298] [Originator@6876 sub=Nfcsvc] NFC server scheduled
YYYY-MM-DDTHH:MM:SSZ info hostd[2101410] [Originator@6876 sub=Libs opID=nbdmode-000000XXXX] Setting NFC log level to 1
YYYY-MM-DDTHH:MM:SSZ warning hostd[2101254] [Originator@6876 sub=Libs opID=nbdmode-000000XXXX] [NFC ERROR]NfcCheckAndReserveMem: Cannot allocate anymore memory as NFC is already using 100663192 and allocating 168 will make it more than the maximum allocated: 100663296. Please close some sessions and try again.


Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

NFC operations are running out of memory. This issue typically arises when the host exhausts its NFC memory due to excessive retries or lingering stale sessions that haven't been cleared.

Resolution

To resolve the issue, we have to increase the session memory for NFC on host by modifying the hostd configuration file. 

In earlier versions,  hostd settings could be modified by editing the configuration file /etc/vmware/hostd/config.xml. However, starting with ESXi 7.0 U2, the service configuration settings are now stored in a dedicated configuration store database accessible by using /bin/configstorecli.

Below are the steps to modify the NFC settings:

1. Import the configuration to a temporary JSON file:
  $ /bin/configstorecli config current get -c esx -g services -k hostd -outfile tmp.json

2. Edit the file:
  $ vi tmp.json

3. By default, maxMemory parameter is set to value 100663296. Set it to 150663296 (set the value as per the environmental requirement) :
  <nfcsvc>
        <path>libnfcsvc.so</path>
        <enabled>true</enabled>
        <maxMemory>100663296</maxMemory>           <================= increased this to 150663296
        <maxStreamMemory>35651584</maxStreamMemory>
  </nfcsvc>

4. Save the changes : Hit Esc -> :wq!

5. Apply the file to the database:
  $ /bin/configstorecli config current set -c esx -g services -k hostd -infile tmp.json

6. Restart hostd service:
  $ /etc/init.d/hostd restart

Additional Information

NFC operations running out of memory indicate that there are concurrent NFC sessions requests coming in for the ESXi host.

It is advised to reduce the number of NFC sessions getting created. For instance, where the NFC operations are the VM replication requests, the number of VM replications can be reduced.