"Error while disabling jobs: disable_jobs(cur_odbc, con_odbc, args.rdbms)" at Stage 2 of migration to vCenter Appliance 6.x/7.x from a vCenter Server 6.x with external MS SQL database

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• When migrating from from a vCenter Server 6.x with external MS SQL database to vCenter Appliance 6.x/7.x, in Stage 2 of the migration, error is seen:

Error disabling jobs: disable_jobs(cur_odbc, con_odbc, args.rdbms) 

• In the /var/log/vmware/upgrade/upgrade-export.log file:

YYYY-MM-DDTHH:MM:SS ERROR upgrade_commands Export failed with {'componentKey': 'com.vmware.vcdb', 'detail': [{'args': [' disable_jobs(cur_odbc, con_odbc, args.rdbms)\n'], 'translatable': 'Error while disabling jobs: %(0)s', 'id': 'vcdb.error.export.disable.jobs', 'localized': 'Error while disabling jobs: disable_jobs(cur_odbc, con_odbc, args.rdbms)\n'}], 'problemId': None, 'resolution': {'localized': 'Read the log files and address the issues experienced while disabling your jobs.', 'translatable': 'Read the log files and address the issues experienced while disabling your jobs.', 'id': 'vcdb.action.disable.jobs.log'}}. Check UpgradeRunner.log log for details. 

• In the Export_com.vmware.vcdb_YYYY_MM_DD_HH_mm.log file (being the date of the migration attempt):

YYYY-MM-DDTHH:MM:SS ERROR vcdb.const Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)
YYYY-MM-DDTHH:MM:SS ERROR __main__ Upgrade Phase 'vcdb:Export' failed. Exception: Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)
YYYY-MM-DDTHH:MM:SS.log:    ret = disable_jobs(**kwargs)
YYYY-MM-DDTHH:MM:SS.log:  File "C:\migA23E.tmp\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\vcdb\vcdb_db_utils.py", line 883, in disable_jobs
YYYY-MM-DDTHH:MM:SS.log:PermanentError: Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)

• In the UpgradeRunner.log file:

YYYY-MM-DDTHH:MM:SS ERROR vcdb.const Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)
YYYY-MM-DDTHH:MM:SS ERROR __main__ Upgrade Phase 'vcdb:Export' failed. Exception: Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)
YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states vcdb:Export: PermanentError: Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)

• In the bootstrap.log file:

YYYY-MM-DDTHH:MM:SS ERROR upgrade_commands Export failed with {'componentKey': 'com.vmware.vcdb', 'detail': [{'args': ['    disable_jobs(cur_odbc, con_odbc, args.rdbms)\n'], 'translatable': 'Error while disabling jobs: %(0)s', 'id': 'vcdb.error.export.disable.jobs', 'localized': 'Error while disabling jobs:     disable_jobs(cur_odbc, con_odbc, args.rdbms)\n'}], 'problemId': None, 'resolution': {'localized': 'Read the log files and address the issues experienced while disabling your jobs.', 'translatable': 'Read the log files and address the issues experienced while disabling your jobs.', 'id': 'vcdb.action.disable.jobs.log'}}. Check UpgradeRunner.log log for details.
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on environment.

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.7.x

Cause

This issue occurs if the necessary database permissions are not available.

Resolution

To resolve the issue, check the required database permissions and restart the migration process.

Process to check the required database permissions:

1. Permission for the DB user used for vCenter Server
   1. Log in to SQL Management Studio
   2. Go to Security > Logins > Select the user used to interact with the vCenter database > right click > select properties
   3. Under the vCenter database (by default vcdb), check that the selected user in the previous step has the db_owner role membership.
      • Note: Database role db_owner is a simple way of assigning permissions for vCenter database user. For more information refer Point 1 in additional info.

2. Permissions for the owner of the SQL jobs in the vCenter database
   1. Log in to SQL Management Studio
   2. Expand Databases > Expand the database of the vCenter (by default vcdb) 
   3. Expand Schemas > Expand the schema in use (by default dbo) > right click Properties > Permissions > View database permissions > Select Effective
   4. Check that the permissions applied to that user/schema are similar to the ones listed in Database Permission Requirements for vCenter Server (Refer Point 2 in Additional info)
3. Re-start the migration process.

Additional Information

1. Prepare the vCenter Server Database by Using the dbo Schema and the db_owner Database Role
   

   Procedure
   Create a database and user for vCenter Server.
   In the master database, create a database for vCenter Server.
   Create a database user for vCenter Server and map it to the vCenter Server and msdb databases.
   For example, to create the database VCDB and user vpxuser, you can run the following script:
   
use master go CREATE DATABASE VCDB ON PRIMARY (NAME = N'vcdb', FILENAME = N'C:\database_path\VCDB.mdf', SIZE = 10MB, FILEGROWTH = 10% ) LOG ON (NAME = N'vcdb_log', FILENAME = N'C:\database_path\VCDB.ldf', SIZE = 1000KB, FILEGROWTH = 10%) COLLATE SQL_Latin1_General_CP1_CI_AS go use VCDB go CREATE LOGIN vpxuser WITH PASSWORD=N'vpxuser!0', DEFAULT_DATABASE=VCDB, DEFAULT_LANGUAGE=us_english, CHECK_POLICY=OFF go CREATE USER vpxuser for LOGIN vpxuser go use MSDB go CREATE USER vpxuser for LOGIN vpxuser go
   
   You now have a Microsoft SQL Server database that you can use with vCenter Server.
   Assign the db_owner role to the vCenter Server database user on both the vCenter Server and msdb databases.
   For example, to assign the db_owner role to the vpxuser user, you can run the following script:
   
use VCDB go sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser' go use MSDB go sp_addrolemember @rolename = 'db_owner', @membername = 'vpxuser' go
   
   Enable database monitoring for the vCenter Server database user.
   For example, to grant database disk size monitoring permissions to the vpxuser user, you can run the following script:
   
   use master go grant VIEW SERVER STATE to vpxuser go GRANT VIEW ANY DEFINITION TO vpxuser go
   
   Results
   When you install vCenter Server, the installer uses the default dbo schema to assign permissions to the db_owner role.

2. vCenter Server requires a database. If you decide to use an external Oracle or Microsoft SQL Server database, when you create the database, you must grant certain permissions to the database user.
   

   Table 1. Microsoft SQL Database Permissions for vCenter Server
   Permission    Description
   GRANT ALTER ON SCHEMA :: [VMW] TO VC_ADMIN_ROLE    Mandatory when you work with SQL Server custom schema.
   GRANT REFERENCES ON SCHEMA :: [VMW] TO VC_ADMIN_ROLE    Mandatory when you work with SQL Server custom schema.
   GRANT INSERT ON SCHEMA :: [VMW] TO VC_ADMIN_ROLE    Mandatory when you work with SQL Server custom schema.
   GRANT CREATE TABLE TO VC_ADMIN_ROLE    Necessary for creating a table.
   GRANT CREATE VIEW TO VC_ADMIN_ROLE    Necessary for creating a view.
   GRANT CREATE PROCEDURE TO VC_ADMIN_ROLE    Necessary for creating a stored procedure.
   GRANT SELECT ON SCHEMA :: [VMW] TO VC_USER_ROLE    Permissions that let you run SELECT, INSERT, DELETE, UPDATE operations on tables which are part of the VMW schema.
   GRANT INSERT ON SCHEMA :: [VMW] TO VC_USER_ROLE
   GRANT DELETE ON SCHEMA :: [VMW] TO VC_USER_ROLE
   GRANT UPDATE ON SCHEMA :: [VMW] TO VC_USER_ROLE
   GRANT EXECUTE ON SCHEMA :: [VMW] TO VC_USER_ROLE    Necessary for running a stored procedure in the db schema.
   GRANT SELECT ON msdb.dbo.syscategories TO VC_ADMIN_ROLE    Necessary for deploying SQL Server jobs.
   These permissions are mandatory only during installation and upgrade and not required after deployment.

   GRANT SELECT ON msdb.dbo.sysjobsteps TO VC_ADMIN_ROLE
   GRANT SELECT ON msdb.dbo.sysjobs TO VC_ADMIN_ROLE
   GRANT SELECT ON msdb.dbo.syssessions TO VC_ADMIN_ROLE
   GRANT SELECT ON msdb.dbo.sysjobactivity TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_add_job TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_delete_job TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_add_jobstep TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_update_job TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_add_jobserver TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_add_jobschedule TO VC_ADMIN_ROLE
   GRANT EXECUTE ON msdb.dbo.sp_add_category TO VC_ADMIN_ROLE
   GRANT VIEW SERVER STATE TO [vpxuser]    Provides access to SQL Server DMV views and sp_lock execution.
   GRANT VIEW ANY DEFINITION TO [vpxuser]    Necessary for providing the user with the privileges to see metadata for SQL Server objects.
   Table 2. Oracle Database Permissions for vCenter Server
   
   Permission    Description
   GRANT CONNECT TO VPXADMIN    Necessary for connecting to the Oracle database.
   GRANT RESOURCE TO VPXADMIN    Necessary for creating a trigger, sequence, type, procedure, and so on.
   By default, the RESOURCE role has the CREATE PROCEDURE, CREATE TABLE, and CREATE SEQUENCE privileges assigned. If the RESOURCE role lacks these privileges, grant them to the vCenter Server database user.

   GRANT CREATE VIEW TO VPXADMIN    Necessary for creating a view.
   GRANT CREATE SEQUENCE TO VPXADMIN    Necessary for creating a sequence.
   GRANT CREATE TABLE TO VPXADMIN    Necessary for creating a table.
   GRANT CREATE MATERIALIZED VIEW TO VPXADMIN    Necessary for creating a materialized view.
   GRANT EXECUTE ON dbms_lock TO VPXADMIN    Necessary for guaranteeing that the vCenter Server database is used by a single vCenter Server instance.
   GRANT EXECUTE ON dbms_job TO VPXADMIN    Necessary during installation or upgrade for scheduling and managing the SQL jobs.
   This permission is not required after deployment.

   GRANT SELECT ON dba_lock TO VPXADMIN    Necessary for determining existing locks on the vCenter Server database.
   GRANT SELECT ON dba_tablespaces TO VPXADMIN    Necessary during upgrade for determining the required disk space.
   This permission is not required after deployment.

   GRANT SELECT ON dba_temp_files TO VPXADMIN    Necessary during upgrade for determining the required disk space.
   This permission is not required after deployment.

   GRANT SELECT ON dba_data_files TO VPXADMIN    Necessary for monitoring the free space while vCenter Server is working.
   GRANT SELECT ON v_$session TO VPXADMIN    View used to determine existing locks on the vCenter Server database.
   GRANT UNLIMITED TABLESPACE TO VPXADMIN    Necessary for granting unlimited tablespace permissions to the vCenter Server database user.
   GRANT SELECT ON v_$system_event TO VPXADMIN    Necessary for checking log file switches.
   GRANT SELECT ON v_$sysmetric_history TO VPXADMIN    Necessary for checking the CPU utilization.
   GRANT SELECT ON v_$sysstat TO VPXADMIN    Necessary for determining the Buffer Cache Hit Ratio.
   GRANT SELECT ON dba_data_files TO VPXADMIN    Necessary for determining the tablespace utilization.
   GRANT SELECT ON v_$loghist TO VPXADMIN    Necessary for checking the checkpoint frequency.
   The privileges on the master database are used to monitor the vCenter Server database. So that, for example, if a certain threshold is reached you can see an alert.

3. Preparing Microsoft SQL Server Database Before Upgrading vCenter Server,
   

   Procedure
   Verify that your database meets the upgrade requirements. If necessary, upgrade the database to a supported version.
   If vCenter Server does not support your database server, perform a database upgrade to a supported version or import your database into a supported version.
   If your existing database is Microsoft SQL Server, and you want to upgrade to a newly supported Microsoft SQL Server database, such as Microsoft SQL Server 2012, upgrade your Microsoft SQL Server database before upgrading vCenter Server.
   If your existing database is Microsoft SQL Server, you do not need to install a new vCenter Server instance.
   When you migrate the database from Microsoft SQL Server 2005 to Microsoft SQL Server 2008 R2-SP2 or later, set the compatibility level of the database to 100.
   Verify that permissions are set correctly.
   Verify that passwords are current and not set to expire soon.
   Verify that JDK 1.6 or later is installed on the physical vCenter Server machine.
   Verify that the sqljdbc4.jar file is added to the CLASSPATH variable on the physical machine where vCenter Server is to be upgraded.
   If the sqljdbc4.jar file is not installed on your system, the vCenter Server installer installs it.
   Verify that your system database source name is using the Microsoft SQL Server Native Client 10 or 11 driver.
   If you choose to remove the DBO role and migrate all objects in the DBO schema to a custom schema, you must grant the required permissions.
   Grant the required permissions to the vCenter Server user in the vCenter Server database.
   Grant the required permissions to the user in the MSDB database.
   Make a full backup of the vCenter Server database.

   Results
   Your database is prepared for the vCenter Server upgrade.

4. Additional information regarding SQL Server Management Studio (SSMS), refer to Microsoft Tutorials similar to this: Tutorial: Connect to and query a SQL Server instance by using SQL Server Management Studio (SSMS)