vCenter Server Upgrade Fails Due to Stopped Content Library Service

Products

• VMware vCenter Server

Issue/Introduction

When attempting to upgrade VMware vCenter Server from version 7.0 Update 3i to a newer version, the upgrade process may fail with an error indicating that the Content Library Service has stopped. This issue prevents the successful completion of the vCenter Server upgrade.

Environment

• VMware vCenter Server
• Content Library Service stopped prior to upgrade attempt, or
• Content Library tables are not present or have invalid data.
  • See the Resolution section for steps for determining if the necessary Content Library tables are present and contain valid data.

Cause

The Content Library Service relies on specific database tables and configurations to function properly. If these tables are missing, have invalid values, or the service was stopped before the upgrade, it can cause the vCenter Server upgrade to fail.

Resolution

Follow these steps to connect to the vCenter Server Appliance, access the PostgreSQL database, and perform the necessary tests to verify the presence and validity of the Content Library database tables.

1. Connect to the vCenter Server Appliance (VCSA) using SSH:
   1. Open an SSH client (e.g., PuTTY, Terminal).
   2. Enter the IP address or hostname of the VCSA.
   3. Provide the root username and password when prompted.
      
      
2. Access the PostgreSQL database:
   • Connect to the PostgreSQL database using the following command:
        /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
     
     
3. Verify the Content Library database tables:
   1. Run the following query to check the existence of the required tables:
            SELECT table_name FROM information_schema.tables WHERE table_schema='vc' AND table_name LIKE 'cl_%';
   2. Review the output to ensure that the necessary Content Library tables are present.
      
      
4. Check the contents of the cl_config table:
   1. Run the following query to view the contents of the cl_config table:
      
      SELECT * FROM cl_config;
      
      
   2. Verify that the database.schema.version value is set correctly.
      
      
5. Verify the existence and contents of the cl_subscription table:
   1. Run the following query to check if the cl_subscription table exists and contains records:
      
      SELECT * FROM cl_subscription;


   2. If records are present:
      • Do not proceed with further steps.
      • Open a technical support case with Broadcom
      • Provide your current vCenter version and build
      • Share the output from the commands above
        
        
   3. If no records are present:
      • The issue may be corruption of the cl_subscription table
      • In that case, follow the directions below under "Recreating the cl_subscription table"
        
        
6. Check the contents of the cl_library table:
   1. Run the following query to view the contents of the cl_library table:
      
SELECT * FROM cl_library;

   2. Review the output to ensure the table contains valid data.
      
      
7. Perform additional tests (optional):
   1. Run the following queries to investigate related tables:
      
      SELECT * FROM cl_libraryitemhistory;
SELECT * FROM cl_libraryItemVersion;
SELECT * FROM cl_security_rule;
SELECT * FROM cl_security_policy_rule;
\d cl_library;
\d cl_subscription;
\d cl_libraryitemhistory;
\d cl_libraryItem;
\d cl_registry;
\d cl_registry_project_member;
      
      
   2. Review the output of each query to gather more information about the Content Library database.
      
      
8. Exit the PostgreSQL database:
   • Type \q and press Enter to exit the PostgreSQL database.
     
     
9. Disconnect from the vCenter Server Appliance:
   • Type exit and press Enter to close the SSH connection.
     
     

Recreating the cl_subscription table
From Step 5 c. above, if no records are present, re-create the cl_subscription table

Please note that the cl_subscription table, by default, contains no records, unless a content library subscription is in place. However, if receiving the error related to this article, we're assuming the cl_subscription table has corrupted and therefore needs to be re-created:

1. Run the command to remove the current cl_subscription table:

DROP TABLE vc.cl_subscription;
   
   
2. Rebuild the vc.cl_subscription table:
   
   CREATE TABLE IF NOT EXISTS vc.cl_subscription (
    id uuid NOT NULL,
    publishedlibraryid uuid NOT NULL,
    subscribedlibraryid uuid NOT NULL,
    subscribedlibraryname character varying(80),
    subscriber_vcenter_hostname character varying(255),
    subscriber_vcenter_https_port integer,
    subscriber_server_guid uuid NOT NULL,
    folderid character varying(80),
    clusterid character varying(36),
    resourcepoolid character varying(36),
    hostid character varying(36),
    networkid character varying(36)
);
   
   
3. Set table owner
   
   ALTER TABLE vc.cl_subscription OWNER TO postgres;
   
   
4. Add primary key constraint
   
   ALTER TABLE ONLY vc.cl_subscription
ADD CONSTRAINT cl_subscription_pk PRIMARY KEY (id);
   
   
5. Add unique constraint
   
   ALTER TABLE ONLY vc.cl_subscription
ADD CONSTRAINT cl_subscription_uk UNIQUE (publishedlibraryid, subscribedlibraryid);


6. Grant permissions

   GRANT ALL ON LANGUAGE plpgsql TO vcgroup;
GRANT ALL ON TABLE vc.cl_subscription TO vcgroup;
GRANT SELECT, INSERT, DELETE, UPDATE ON TABLE vc.cl_subscription TO cls;

7. If any issues with these commands, open a case with Broadcom technical support.