vCenter Tasks fail and EventManager fails with the error vmodl.fault.InvalidArgument

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
You may see the following symptoms:

• Tasks are failing or hanging in vCenter
• Requests to EventManager are failinng with the error "vmodl.fault.InvalidArgument"
• You see Event logs in the journalctl logs which have a negative number for the event (in bold):

Jun 29 13:33:58 vcenter.example.com vpxd[3051]: Event [-969673591] [1-1] [2022-06-
29T13:33:58.325453Z] [vim.event.UserLoginSessionEvent] [info] [DOMAIN\User] [] [-969673591] [User DOMAIN\[email protected] logged in as pyvmomi Python/3.6.12 (Linux; 4.12.14-122.60-default; x86_64)]

• You see the following in the vpxd logs, also showing negative value (called out in bold):

2022-07-11T16:17:21.858Z info vpxd[08330] [Originator@6876 sub=vpxLro opID=3dbebf4f] [VpxLRO] -- FINISH lro-95693867
2022-07-11T16:17:21.858Z info vpxd[08330] [Originator@6876 sub=Default opID=3dbebf4f] [VpxLRO] -- ERROR lro-95693867 -- EventManager -- vim.event.EventManager.QueryEvent: vmodl.fault.InvalidArgument:
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = "filter"
-->    msg = ""
--> }
--> Args:
-->
--> Arg filter:
--> (vim.event.EventFilterSpec) {
-->    entity = (vim.event.EventFilterSpec.ByEntity) {
-->       entity = 'vim.VirtualMachine:vm-1234560',
-->       recursion = "self"
-->    },
-->    time = (vim.event.EventFilterSpec.ByTime) null,
-->    userName = (vim.event.EventFilterSpec.ByUsername) null,
-->    eventChainId = -968541408,  <<----  (NEGATIVE VALUE)
-->    alarm = <unset>,
-->    scheduledTask = <unset>,
-->    disableFullMessage = <unset>,
-->    category = <unset>,
-->    type = <unset>,
-->    tag = <unset>,
-->    eventTypeId = (string) [
-->       "vim.event.TaskEvent"
-->    ],
-->    maxCount = <unset>,
-->    order = <unset>
--> }

• You see the following error:

SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007eff2c0e7178, h:101, <TCP '127.0.0.1 : 41556'>, <TCP '127.0.0.1 : 80'>>), /sdk>, method: createTask; code: 500(Internal Server Error)

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.x

Cause

This could be caused by an overrun on the event ID or chain ID in the vCenter database.

Resolution

Refer to the solution in the following KB to delete old tasks and events and reset the counter:

Delete old tasks, events, statistics data and sequences reset in vCenter Server 5.x, 6.x and 7.x

 

Note: You may want to review the frequency of events occurring for the vCenter to ensure that there is not an issue such as an account which is logging in and out frequently, causing many events.