Adding a host to VMware vCenter Server fails with the error: failed to get DVS state from vmkernel status (bad0006) = Limit exceeded

Products

• VMware

Issue/Introduction

Symptoms:

• You are unable to add a ESXi/ESX host to vCenter Server.
• The VMware vCenter Server Agent ( vpxa) fails.
• The messages.log (located at /var/log/messages.log) contains entries similar to:
  
  FetchPortState call failed: caught vim.fault.PlatformConfigFault exception
  May 17 14:15:24 vmkernel: 90:15:17:39.429 cpu12:37090547)User: 2428: wantCoreDump : vpxa -enabled : 1
  May 17 14:15:24 vmkernel: 90:15:17:39.571 cpu12:37090547)UserDump: 1485: Dumping cartel 37885140 (from world 37090547) to file /var/core/vpxa-zdump.003 ...
  May 17 14:15:27 Hostd: [2011-05-17 14:15:27.248 58840B90 verbose 'vm:/vmfs/volumes/4cfe87b5-ebf6cd41-a2c1-0026b9668f7b/XDVL03-2/XDVL03-2.vmx'] Tools guest daemon status changed to: 2
  May 17 14:15:27 Hostd: [2011-05-17 14:15:27.888 58A81B90 verbose 'vm:/vmfs/volumes/4cfe87b5-ebf6cd41-a2c1-0026b9668f7b/XDVL03-2/XDVL03-2.vmx'] Tools guest daemon status changed to: 1
  May 17 14:15:28 Hostd: [2011-05-17 14:15:28.715 FFCFAB90 verbose 'vm:/vmfs/volumes/4cfe87b5-ebf6cd41-a2c1-0026b9668f7b/XDCITY55/XDCITY55.vmx'] Tools guest daemon status changed to: 2
  May 17 14:15:29 vmkernel: 90:15:17:44.387 cpu4:37090547)UserDump: 1587: Userworld coredump complete.
  May 17 14:15:29 Hostd: [2011-05-17 14:15:29.754 FFE40B90 warning 'Proxysvc Req00032'] Server localhost:8089 disconnected with a response pending.
  (TCP(local=10.1.22.208:4507, peer=10.88.88.112:443))
  
  
• The hostd.log (located at /var/log/hostd.log) contains entries similar to:
  
  2011-05-17 15:19:22.264 FFE40B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
  [2011-05-17 15:20:09.004 FFE40B90 verbose 'DvsManager'] PersistAllDvsInfo called
  [2011-05-17 15:20:09.007 FFE40B90 warning 'NetworkProvider'] Unable to persist dvs config : Failed to get DVS state from vmkern
  el Status (bad0006)= Limit exceeded
  [2011-05-17 15:20:09.007 FFE40B90 info 'DvsManager'] Error while saving DVS information for all dvses: vim.fault.PlatformConfig
  Fault
  [2011-05-17 15:20:09.009 FFE40B90 warning 'NetworkProvider'] Error getting dvport list for dvs 0d 9a 05 50 67 70 68 d0-fa 4d 4f
  50 55 ed b9 c6: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded
  
  
• When you run the #esxcfg-vswitch -l command, the output is similar to:
  
  DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
  dvsname 256 47 256 1500 vmnic0
  
  DVPort ID In Use Client
  Listing failed for DVSwitch: DvsPortset-4, Error: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded
  DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
  dvsname 256 5 256 1500 vmnic6
  
  DVPort ID In Use Client
  Listing failed for DVSwitch: DvsPortset-5, Error: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded
  DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
  dvsname 256 2 256 1500 vmnic1
  
  DVPort ID In Use Client
  Listing failed for DVSwitch: DvsPortset-6, Error: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded
  DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
  dvsname 256 5 256 9000 vmnic7
  
  DVPort ID In Use Client
  Listing failed for DVSwitch: DvsPortset-1, Error: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded
  DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
  dvsname 256 4 256 1500 vmnic11
  
  DVPort ID In Use Client
  Listing failed for DVSwitch: DvsPortset-2, Error: Unable to Get DVPort list ; Status(bad0006)= Limit exceeded

Cause

This issue occurs due to the race condition of modifying vDS properties between port disable and other threads. The sws[0]->ports[14]-> data single direction list can be operated by two different threads by calling the function DVSData_Update().

Resolution

This issue is resolved in ESXi/ESX 4.1 Update 2. For more information about this version, see the ESXi 4.1 Update 2 Release Notes. You can download the latest release from the VMware Download Center.

To work around this issue when you do not want to upgrade, reboot the ESXi/ESX host.