SNMP daemon fails with the error: snmpd: load_fib:189 out of memory

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When running ESXi 5.1 and you have configured Simple Network Management Protocol (SNMP), you experience these symptoms:

• When enabling the SNMP daemon (SNMPD) on the ESXi host, the process fails after few minutes.
• In the syslog.log file, located at /var/log/, you see entries similar to:
  
  14:01:01Z watchdog-snmpd: '/sbin/snmpd ++min=0,group=host/vim/vmvisor/snmpd,securitydom=0' exited after 1029 seconds 139
  
  
• In the hostd.log file, located at /var/log/, you see entries similar to:
  
  12:22:42.413Z [FF9C0D20 error 'Snmpsvc'] OpenAgentCtrl:connect(/var/run/snmp.ctl) failed 2 No such file or directory
  
  
• When enabling SNMP debug logging with the esxcli system snmp set –l debug command, you see entries similar to:

20:08:08Z snmpd: load_fib: v4 routes: 4 v6 routes: 1127056

20:08:08Z snmpd: load_fib:189 out of memory

Note: To return to default info logging, run this command:

esxcli system snmp set –l info

Cause

This issue occurs when IPv6 support is disabled and the SNMP agent stops due to an uninitialized variable.

Resolution

This issue is resolved in VMware ESXi 5.1, Patch Release ESXi510-201310001 (2053402). You can download the latest release of ESXi 5.1 from VMware Downloads.

To work around this issue when you are unable to upgrade, disable the SNMP agent or enable IPv6 support on the host.

Note: In ESXi 5.1 and later, IPv6 is enabled by default.

To disable the SNMP agent on the host:

1. Log in to the ESXi host as the root user.
2. Run this command:
   
   esxcli system snmp set --enable false
   
   
3. Reboot the host to apply the changes.

To enable IPv6, use vSphere Client or log in to the ESXi host.

To enable IPv6 using the vSphere Client:

1. Open the vSphere Client.
2. Navigate to Inventory > Hosts and Clusters.
3. Click to select the host and click the Configuration tab.
4. Click Networking under Hardware.
5. Click vSphere Standard Switch, and then click Properties.
6. Select Enable IPv6 support on this host system and click OK.
7. Reboot the host.

To enable IPv6 using the vSphere Web Client:

1. Open the vSphere Web Client.
2. Click Hosts and Clusters.
3. Select the ESXi host from the list on the left.
4. Click the Manage tab.
5. Click Networking > Advanced.
6. Click Edit.
7. Use the IPv6 support dropdown to enable IPv6 on this vSwitch if it is not enabled and then click OK.
8. Reboot the host.

To enable IPv6 support on the ESXi host command line:

1. Log in the ESXi host as the root user.
2. To enable IPv6 support, run one of these commands:
   
   esxcfg-vmknic -6 true | false
   
   or
   
   esxcli network ip set -e true | false
   
   
3. Reboot the host.

Additional Information

In the syslog.log file example in the Symptoms section, the watchdog utility shows the failure occurs 1029 seconds or 17 minutes after restart. 139 indicates a Signal 11 (SEGV) failure.

For more information about SNMP and vSphere 5.1, see the Monitoring Networked Devices with SNMP and vSphere section in the VMware vSphere 5.1 Documentation Center.

VMware ESXi 5.1, Patch Release ESXi510-201310001

Impact/Risks:
Monitoring ESXi via SNMP becomes sporadic as the watchdog utility tries to restart SNMP.