Snapshot Deletion Job not working through Automation Central in Aria Operations
search cancel

Snapshot Deletion Job not working through Automation Central in Aria Operations

book

Article ID: 448164

calendar_today

Updated On:

Products

VCF Operations VMware Aria Operations (formerly vRealize Operations) 8.x

Issue/Introduction

  • Aria Operations Automation Central - Scheduled Snapshot Deletion Job is not working after executing jobs once.
    Even after rebooting all the cluster nodes (primary/replica/data) might fix the issue temporary, but the scheduled deletion job continues to fail.

  • No communication issues observed from the associative vCenter over port 443, no issues from collection and connectivity observed
  • Logs shows no error and actionScheduler agent (automation central) working the way it is suppose to

actionScheduler-<uuid>.log -

YYYY-MM-DDTHH:MM:SS,816+0000 INFO  [DistTaskActionScheduler] com.vmware.vcops.analytics.action.schedule.ActionScheduler.doRun - Start processing action schedules at <epoch-time>
YYYY-MM-HH:MM:SS,006+0000 INFO  [DistTaskActionScheduler] com.vmware.vcops.analytics.action.schedule.ActionScheduler.doRun - Finish processing action schedules at <epoch-time>
YYYY-MM-HH:MM:SS,985+0000 INFO  [Analytics Main Thread] com.vmware.vcops.analytics.action.schedule.ActionScheduler.init - Action Scheduler is scheduled to start at Tue Apr 07 HH:MM:SS UTC 2026, and repeat every 5 min
YYYY-MM-DDTHH:MM:SS,689+0000 INFO  [SyncResourceWorker-0] com.vmware.vcops.analytics.action.schedule.jobhistory.JobHistoryShardingModule.synchronizeData - synchronizing job Id=########-####-####-####-############ to ########-####-####-####-############
YYYY-MM-DDTHH:MM:SS,357+0000 INFO  [Function Execution Processor21] com.vmware.vcops.analytics.action.schedule.jobhistory.JobHistoryShardingServer.deleteJobHistory - delete job history older than or equal to <epoch-time>



  • The analytics logs show the initial failure to dispatch the email notification:
    YYYY-MM-DDTHH:MM:SS,577+0000 ERROR [pool-44-thread-108] com.vmware.vcops.platform.notification.email.EmailConfig.<init> - Service Credential must be provided when 'Requires Authentication' is checked.
YYYY-MM-DDTHH:MM:SS,578+0000 ERROR [pool-44-thread-108] com.vmware.vcops.analytics.action.schedule.ActionScheduler.run - Unexpected error while sending notification for schedule with name=Delete Snapshots that are older than 7 days_test_m02_h02
java.lang.IllegalArgumentException: Service Credential must be provided when 'Requires Authentication' is checked.
at com.vmware.vcops.platform.notification.email.EmailConfig.<init>(EmailConfig.java:93)
at com.vmware.vcops.platform.notification.email.EmailConfig.fromTransmissionConfig(EmailConfig.java:193)
at com.vmware.vcops.controller.data.utils.action.notification.NotificationEmailSender.initEmailConfig(NotificationEmailSender.java:128)
at com.vmware.vcops.controller.data.utils.action.notification.NotificationEmailSender.<init>(NotificationEmailSender.java:56)
at com.vmware.vcops.analytics.action.schedule.job.JobNotificationThread.run(JobNotificationThread.java:80)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

  • Subsequent logs show the scheduler skipping or flagging the job because it was detected as already active:

    YYYY-MM-DDTHH:MM:SS,708+0000 INFO [DistTaskActionScheduler] com.vmware.vcops.analytics.action.schedule.ActionScheduler.enqueueActionSchedule - Schedule ScheduleDefinition{scheduledTask=ScheduledTask

     

  • Unknown macro: {taskKey=CustomActionTaskKey Unknown macro}

    com.vmware.vcops.platform.api.model.schedule.task.CustomActionTaskKey@########}, notificationConfigs=[com.vmware.vcops.platform.api.model.schedule.notification.EmailNotificationConfig@########], nextNotificationAt=<epoch-time>} Schedule [uuid=########-####-####-####-############ name=Delete Snapshots that are older than 7 days_test_m02_h02 type=DAILY_SCHEDULE startOn=<epoch-time>] is running. Skipping

     

Environment

VMware Aria Operations 8.18.x

Cause

This issue can cause where the automated jobs scheduled from automation central in aria operations are failing to progress due to a mismatch in the SMTP/Notification plugin configuration. If "Required Authentication" is checked, "Credential type" cannot be set to "No Credential".
Because of this issue, the Automation Central job is stuck in the notification sending process.
Even if the email plugin's configuration is corrected, existing scheduled instances may remain in an unresolved state because they are tied to the original run context.

Resolution

To fix this issue, refer the following steps - 

  • Clone the Affected Job: Create a duplicate of the stuck scheduled job. This ensures a clean initialization context
  • Disable Notifications: In the newly cloned job, temporarily disable the notification/email plugin.
  • Monitor for Two Cycles: Allow the new job to run for at least two consecutive scheduled cycles

Or

To fully resolve this issue and prevent similar state-locking failures in the future, we recommend the following actions:

  • Correct the Email Plugin Configuration (Customer Action)
  • Review and fix the email plugin configuration within the customer's local environment. Once verified, re-enable email notifications on the Automation Central job.

Additional Information

The above isolation test will allow us to confirm if the core job execution logic completes successfully when entirely decoupled from the notification layer. If the cloned job runs successfully for two cycles, it confirms the issue was strictly tied to the notification config.
Once confirmed, you can re-enable notifications on a fresh job instance using the corrected SMTP/Notification plugin and then remove the original job.

Powered by Wolken Software