Checking the status of vCenter Server performance rollup jobs
search cancel

Checking the status of vCenter Server performance rollup jobs

book

Article ID: 306121

calendar_today

Updated On:

Products

Support Offerings VMware vCenter Server

Issue/Introduction

This article provides steps to check the status of the vCenter Server performance rollup jobs.
 
Note: Environments using vPostgres and SQL Express databases do not use rollup jobs. The stored procedures are called directly from the vpxd process.

You need to check the status of the vCenter Server performance rollup jobs if:
  • After certain tasks, such as maintenance or upgrades, you have to verify that the vCenter Server performance rollup jobs are working.
  • You see a warning/alert for the vCenter Server database in the vCenter Service Status screen indicating that the jobs are not running.


Symptoms:
  • Within the VMware Orchestrator client you see the NSX workflow, Create NSX Endpoint has failed
  • When you validate the workflow Create NSX Endpoint, it reports that there is a missing workflow
  • You see the error:

    Error executing vCenter Orchestrator workflow: Error executing REST call
     
  • When attempting to collect the inventory for NSX using the Network and Security Inventory within a Compute Resource, the task fails and the monitoring log contains entries similar to:

    Workflow 'vSphereVCNSInventory' failed with the following exception: Error executing vCenter Orchestrator workflow: Error executing REST call.
    Stack trace shows: at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout) at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions) at DynamicOps.VCNSModel.Activities.VcoNsxWorkflowInvoker.Execute[T](IDictionary`2 inputParameters, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetVcoNsxLogOn(LogOnSpec vcoNsxLogOnSpec, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.GetVcoNsxConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.vSphereVCNSInventory_CompiledExpressionRoot_
    TypedDataContext3_ForReadOnly.__Expr21Get() at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext) at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context) at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity) at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor) at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate) at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)


Environment

VMware vCenter Server 5.1.x
VMware vCenter Server 6.0.x
VMware vCenter Server 5.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server 4.0.x
VMware VirtualCenter 2.5.x
VMware vCenter Server Appliance 5.0.x
VMware vCenter Server 4.1.x

Cause

  • Within the VMware Orchestrator client you see the NSX workflow, Create NSX Endpoint has failed
  • When you validate the workflow Create NSX Endpoint, it reports that there is a missing workflow
  • You see the error:

    Error executing vCenter Orchestrator workflow: Error executing REST call
     
  • When attempting to collect the inventory for NSX using the Network and Security Inventory within a Compute Resource, the task fails and the monitoring log contains entries similar to:

    Workflow 'vSphereVCNSInventory' failed with the following exception: Error executing vCenter Orchestrator workflow: Error executing REST call.
    Stack trace shows: at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout) at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions) at DynamicOps.VCNSModel.Activities.VcoNsxWorkflowInvoker.Execute[T](IDictionary`2 inputParameters, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetVcoNsxLogOn(LogOnSpec vcoNsxLogOnSpec, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.GetVcoNsxConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.vSphereVCNSInventory_CompiledExpressionRoot_
    TypedDataContext3_ForReadOnly.__Expr21Get() at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext) at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context) at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity) at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor) at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate) at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

Resolution

In MS SQL Server

In MS SQL server 2005/2008/2008 R2/2012/2014 Full Edition, the jobs reside in the SQL Server Agent.
 
To check the status:
 
Note: Ensure that the SQL Agent service is running. If the SQL Agent service is not running, the performance rollup jobs are not executed.
  1. Launch SQL Management Studio.
  2. Connect to the SQL instance where the vCenter Server database resides.
  3. Expand SQL Server Agent within which the jobs reside.
  4. Locate the Past Day Stats rollup job.
  5. Right-click the job and click View History. The status of the job and other details regarding the schedule are displayed.
  6. Repeat the steps for past week rollup and past month rollup.
 
In MS SQL server 2005/2008/2008 R2 Express Edition, there is no SQL Agent to run the jobs. Instead, the VMware VirtualCenter Server service calls the stored procedures for performing the rollup directly. When the vpxd is running, the calls to run the stored procedures are automatically executed.

In Oracle 10g, 11g, and 12c

If you have a query tool such as SQLPLUS, connect to the Oracle database and execute this SQL statement to check next lastdate and nextdate of job:

SELECT job, to_char(last_date, 'mm/dd/yyyy hh:mi pm') lastdate,to_char(next_date, 'mm/dd/yyyy hh:mi pm') nextdate, failures, broken, what
FROM user_jobs
ORDER BY next_date;

You see output similar to:

JOB LASTDATE NEXTDATE FAILURES B WHAT

2 <Date> <time> 0 N begin stats_rollup1_proc; Purge_Stat1_proc; end;

3 <Date> <time> 0 N begin stats_rollup2_proc; Purge_Stat2_proc; end;

4 <Date> <time> 0 N begin stats_rollup3_proc; Purge_Stat3_proc; end;</time></time></time>

The output shows the last time the job ran and if the job is broken. Oracle sets the next run date to a date far in the future after multiple failures. The column B in the output refers to broken jobs. In this example, N (No) broken jobs was returned.
 

For DB2 v9.5 and v9.7

To verify if the performance rollup jobs exist in the vCenter Server database running on DB2:
  1. Connect to the machine hosting the DB2 database.
  2. Launch the DB2 Control Center.
  3. Connect to the vCenter Server database as a database user.
  4. Launch a DB2 command editor from the Control Center and run this SQL statement:

    SELECT * from SYSTOOLS.ADMIN_TASK_LIST
     
  5. Verify and ensure that you can see these three STATS rollup jobs:

    * STATS_ROLLUP_JOB1_VCX
    * STATS_ROLLUP_JOB2_VCX
    * STATS_ROLLUP_JOB2_VCX

    Note: These are the performance rollup jobs. You may also see additional jobs.
     
  6. To check the status of the jobs, run this SQL statement:

    SELECT name, begin_time, end_time, status FROM systools.admin_task_status ORDER BY begin_time DESC;

    You see output similar to:

    NAME BEGIN_TIME END_TIME STATUS
    STATS_ROLLUP_JOB1_VCX YYYY-MM-DD-01.55.00.843000 YYYY-10-05-01.57.40.296000 COMPLETE
    STATS_ROLLUP_JOB2_VCX YYYY-10-05-01.55.00.855000 YYYY-10-05-01.56.37.407000 COMPLETE
    STATS_ROLLUP_JOB3_VCX YYYY-10-05-01.55.00.859000 YYYY-10-05-01.55.37.406000 COMPLETE

    In this sample output, the jobs were successfully marked as COMPLETE. If any issues were encountered the status appears as FAILED.

vCenter Server Appliance

To verify if the performance rollup jobs exist in the vCenter Server Appliance running on DB2:

Note: This process applies for vCenter Server Appliance 5.0 GA and earlier. Later versions of the vCenter Server Appliance utilize a PostGreSQL.
 
  1. Start a SSH session to the vCenter Server Appliance or any other Linux system hosting the vCenter Server DB running on DB2. When you are prompted to log in, use the user root and the password (for example, you use the default user/password for vCenter Server Appliance:

    User: root
    Password: vmware (default)
     
  2. To connect to the current instance of the DB2 manager:

    VCSA-50:~ # su - db2inst1
    db2inst1@VCSA-50:~> db2
     
  3. To show DB2 profile registry variables that are currently set, run this command:

    db2inst1@VCSA-50:~> db2set -all
     
  4. The prerequisites for the database are:

    db2set DB2_ATS_ENABLE_YES
    db2set DB2_SKIPINSERTED=ON
    db2set DB2_EVALUNCOMMITTED=ON
    db2set DB2_SKIPDELETED=ON
     
  5. To check if the scheduler is enabled, verify if DB2SET DB2_ATS_ENABLE is enabled.
  6. To enable the scheduler run this command in the DB2 command window:

    db2set DB2_ATS_ENABLE=YES
     
  7. To verify the jobs exist you have to run query against the DB. Switch the user to db2inst1 with the commands:

    VCSA-50:~ # su - db2inst1
    db2inst1@VCSA-50:~> db2

    To connect to the vCenter Server database, run the command:

    db2 => connect to VCDB

    To set the schema to vCenter Server, run the command

    db2 => set schema vc
    SELECT name from SYSTOOLS.ADMIN_TASK_LIST

    Verify and ensure that you can see these three STATS rollup jobs:

    * STATS_ROLLUP_JOB1_VCX
    * STATS_ROLLUP_JOB2_VCX
    * STATS_ROLLUP_JOB2_VCX

    Note: These are the performance rollup jobs. You may also see other additional jobs.
     
  8. To check the status of the jobs, run this SQL statement:

    SELECT name, begin_time, end_time, status FROM systools.admin_task_status ORDER BY begin_time DESC ;

    You see output similar to:

    NAME BEGIN_TIME END_TIME STATUS
    STATS_ROLLUP_JOB1_VCX YYYY-10-05-01.55.00.843000 YYYY-10-05-01.57.40.296000 COMPLETE
    STATS_ROLLUP_JOB2_VCX YYYY-10-05-01.55.00.855000 YYYY-10-05-01.56.37.407000 COMPLETE
    STATS_ROLLUP_JOB3_VCX YYYY-10-05-01.55.00.859000 YYYY-10-05-01.55.37.406000 COMPLETE


Workaround:
This is for testing purpose.

Additional Information



Impact/Risks:
  • Within the VMware Orchestrator client you see the NSX workflow, Create NSX Endpoint has failed
  • When you validate the workflow Create NSX Endpoint, it reports that there is a missing workflow
  • You see the error:

    Error executing vCenter Orchestrator workflow: Error executing REST call
     
  • When attempting to collect the inventory for NSX using the Network and Security Inventory within a Compute Resource, the task fails and the monitoring log contains entries similar to:

    Workflow 'vSphereVCNSInventory' failed with the following exception: Error executing vCenter Orchestrator workflow: Error executing REST call.
    Stack trace shows: at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout) at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions) at DynamicOps.VCNSModel.Activities.VcoNsxWorkflowInvoker.Execute[T](IDictionary`2 inputParameters, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetVcoNsxLogOn(LogOnSpec vcoNsxLogOnSpec, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Activities.VcoNsxHelper.GetVcoNsxConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId) at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.vSphereVCNSInventory_CompiledExpressionRoot_
    TypedDataContext3_ForReadOnly.__Expr21Get() at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext) at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context) at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity) at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor) at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate) at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)


Powered by Wolken Software