vCenter Server /storage/log filling up due to many vpxd logs contain API calls referencing non-existent objects, such as "dvportgroup".

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Log partition /storage/log is growing rapidly and/or is running full
• Checking /storage/log/vmware/vpxd/vpxd.log there are a lot of messages referring to a non-existent object of type dvportgroup, like:
  

  YYYY-MM-DDTHH:MM:SS error vpxd[11011] [Originator@6876 sub=Default opID=xxxxxx] [VpxLRO] -- ERROR lro-123456789 -- 1xxxxxxxxxxxxxxxxxx9(7xxxxxxxxxxxxxxxxxxxxxxxxxxx7) -- propertyCollector -- vmodl.query.PropertyCollector.retrievePropertiesEx: :vmodl.fault.ManagedObjectNotFound
  --> Result:
  --> (vmodl.fault.ManagedObjectNotFound) {
      --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = <unset>,
  --> obj = 'vim.dvs.DistributedVirtualPortgroup:dvportgroup-1234'
  --> msg = ""
  --> }
  --> Args: 
  [...]

Environment

VMware vCenter Server 8.x
VMware vCenter Server 7.x

Cause

These errors typically appear when a API call is requesting information about an object which does no longer exist in vCenter. This API call can be from within the vCenter appliance or from a different product or external source. To identify the originating user, we can correlate the Session ID which has triggered this error with the list of sessions and their corresponding username and IP address.

Depending on the originating software solution, these requests can be sent with high intensity, causing many errors to be written and log partition filling up even with log rotation in place.

Resolution

To identify the source of the errors, we can do following:

1. Find a log entry in /storage/log/vmware/vpxd/vpxd.log with the error message outlined above in the issue introduction. For example:

YYYY-MM-DDTHH:MM:SS error vpxd[11011] [Originator@6876 sub=Default opID=xxxxxx] [VpxLRO] -- ERROR lro-123456789 -- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx(7xxxxxxxxxxxxxxxxxxxxxxxxxxx7) -- propertyCollector -- vmodl.query.PropertyCollector.retrievePropertiesEx: :vmodl.fault.ManagedObjectNotFound
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
    --> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> obj = 'vim.dvs.DistributedVirtualPortgroup:dvportgroup-1234'
--> msg = ""
--> }
--> Args: 
[...]

2. In above log entry we do see an attempt of retrieving information about the object dvportgroup-1234, which is a virtual portgroup of a Virtual Distributed Switch. Noteworthy here, is the generated UUID in red: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Please take note of this UUID, this is the vpxd user session UUID.

3. As the next step, please login to vCenter via SSH.

4. Once logged into vCenter, we can check the vpxd-profiler.log log file on vCenter for this particular UUID above, by using following command:

# cat /var/log/vmware/vpxd/vpxd-profiler.log | grep ClientIP | awk -F"/" '{print $5,$6,$7}' | sort | uniq -c | grep "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
2430 Id='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' Username='VSPHERE.LOCAL\Test-User' ClientIP='10.0.0.10'

As seen above output, we can identify that above error is caused due as part of an API request of a specific user session, correlating to user "VSPHERE.LOCAL\Test-User" from IP "10.0.0.10".

5. Please identify the software solution sending this API calls, based with above identified username and ClientIP. Contact the corresponding software vendor if needed.

Additional Information

This behavior has occasionally been observed, for example, in environments where Dell PowerProtect Data Manager software is in use. Rebooting Dell's PowerProtect appliance or vCenter has, in some cases, provided a temporary resolution. For a thorough analysis, questions and/or a permanent solution, we recommend reaching out to the respective software vendor for further assistance.

The issue is supposed to be addressed in Dell PowerProtect Data Manager 19.18.0-23.