Failed or queued VMware vSphere Lifecycle Manager Check Notification tasks in vCenter Server 7.x

Products

VMware vCenter Server

Issue/Introduction

This article provides a way to remove the invalid tasks by means of editing the vCenter Server database.

Symptoms:

Check Notification tasks are triggered in vCenter Server 7.x
The tasks cannot be edited or removed
The user gets check notification emails
The vCenter Server was upgraded from 6.7.x to 7.0.x.
LCM hourly check new notification task is queued and never completes
LCM logs show similar

[YYYY-MM-DD HH:MM:SS 'InventoryMonitor' 140166601914112 INFO] [InventoryMonitor, 497] ProcessUpdate, Enter, Update version := 9
[YYYY-MM-DD HH:MM:SS 'InventoryMonitor' 140166601914112 INFO] [InventoryMonitor, 714] Event value type: vim.event.ScheduledTaskStartedEvent
[YYYY-MM-DD HH:MM:SS 'InventoryMonitor' 140166601914112 INFO] [InventoryMonitor, 1614] UpdateHandler: Last updateSet
[YYYY-MM-DD HH:MM:SS 'ScheduledTaskMgr' 140165734311680 WARN] [scheduledTaskMgr, 1303] Custom fields have been modified by others.
[YYYY-MM-DD HH:MM:SS 'JobDispatcher' 140166602180352 DEBUG] [JobDispatcher, 417] The number of tasks: 0
[YYYY-MM-DD HH:MM:SS 'SessionAuthData' 140166600582912 INFO] [vciSessionAuthData, 74] Trying to get current VC connection for session [526db9a1-de6a-5b8b-43a5-419027b5866b]; user: com.vmware.vim.eam
[YYYY-MM-DD HH:MM:SS 'Activation' 140166600582912 INFO] [activationValidator, 801] Calling VC
[YYYY-MM-DD HH:MM:SS 'Activation' 140166600582912 INFO] [activationValidator, 803] Done, checking privileges
[YYYY-MM-DD HH:MM:SS 'Activation' 140166600582912 INFO] [activationValidator, 805] Done

vmware-vum-server.log shows

YYYY-MM-DD HH:MM:SS error vmware-vum-server[10565] [Originator@6876 sub=ScheduledTaskMgr] [scheduledTaskMgr 2248] Err getting action data from database for key: schedule-902 Error: No record - VCIDB ERROR: Row with key schedule-902 not found in table VCI_TEXTFILES

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.x

Cause

The "VMware vSphere Lifecycle Manager Check Notification" task has been deprecated from the 7.0 release. The vSphere Client does not have the option to edit this task.

In 7.0 U2, the UI to set the check-notification schedule is removed. However, on the upgrade from 6.7 to 7.0, this task and schedule may remain in the DB. This creates an issue, as this cannot be edited from UI and keeps sending emails.

Resolution

NOTE: Please ensure that a fresh backup or offline snapshot of the vCenter was created before proceeding with the steps below. In case the vCenter is part of a Linked Mode replication setup, please be aware that all members of the Linked Mode group need to be backed up/snapshotted.

Connect to vCenter Server Appliance 7.0 using SSH.
Run the shell command to switch to the BASH Shell:
```
# shell
```

Stop the VMware Update Manager Service:

# service-control --stop vmware-updatemgr

Connect to the update manager database, using the following command:

# PGPASSWORD=$(cat /usr/lib/vmware-updatemgr/bin/configvalues.txt | grep "db_password" | awk -F'(db_password=)' '{ print $2 }') /opt/vmware/vpostgres/current/bin/psql -U vumuser -d VCDB

Run the query below to delete the entry causing the tasks:

# delete from vci_textfiles where keystr='integrity.NewUpdateConfigSpec_Notification';

The output should look like this:

VCDB=> delete from vci_textfiles where keystr='integrity.NewUpdateConfigSpec_Notification';
DELETE 1

Run "\q" to leave psql then start the VMware Update Manager service:
```
# service-control --start vmware-updatemgr
```

Workaround:

Alternatively, if the method above fails, the task can also be removed from the vCenter task scheduler. Please be aware that this method however does require a restart of the vCenter Server service (VPXD), and plan accordingly:

list the current scheduled tasks with the following query:

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select scheduledtask_id,name,description from vpx_scheduledtask;"

look through the list and find the task ID for the check notifications task

run the following 3 queries to remove the task (replace <id> with the found in step 2):

# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "delete from vpx_sched_action where scheduledtask_id = <id>;"
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "delete from vpx_sched_scheduler where scheduledtask_id = <id>;"
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "delete from vpx_scheduledtask where scheduledtask_id = <id>;"

restart the vCenter Server service (vpxd) and the Update Manager:

# service-control --restart vmware-vpxd
# service-control --restart vmware-updatemgr

Additional Information

Impact/Risks:
Requires restart of Update Manager service, or of the vCenter Server service, if the alternative steps are chosen.