"An error occurred while starting service statsmonitor", vCenter Server upgrade to 8.0 fails due to a race condition in Statsmonitor service

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server upgrade fails while starting Statsmonitor service with below error message

Failed to start Appliance Stats Monitoring service
An error occurred while starting service 'statsmonitor'

• VCSA CLI log file workflow_xxxxxxxxx/input/taskflow.log will show similar to below snippets :

2023-04-27 19:11:05,576 [loggable.py:102]: ==========VCSA Deployment Progress Report==========
        Task: Run firstboot scripts.(FAILED 84/100)    - Starting VMware Appliance Monitoring Service...
        Error: An error occurred while starting service 'statsmonitor'
        Error: Failed to start Appliance Stats Monitoring service.
        
2023-04-27 19:11:05,576 [loggable.py:102]: The VCSA deployment has failed

• Statsmonitor firstboot stdout log /var/log/firstboot/statsmonitor_firstboot.py_xxxx_stdout.log on VCSA will show similar to below snippets :

2023-04-27T19:10:45.641Z  Starting statsmonitor
2023-04-27T19:10:45.865Z  statsmonitor firstboot failed due to <cis.componentStatus.ErrorInfo object at 0x7f9146e21b90>.

• Statsmonitor firstboot stderr log /var/log/firstboot/statsmonitor_firstboot.py_xxxx_stderr.log on VCSA will show similar to below snippets :

2023-04-27T19:10:45.859Z  ERROR starting statsmonitor rc: 2, stdout: , stderr: Start service request failed. Error: Service crashed while starting

• Statsmonitor service log /var/log/vmware/applmgmt/StatsMonitor.log will show similar to below snippets:

2023-04-27T19:10:45.835Z info StatsMonitor[47911] [Originator@6876 sub=StatsMonitor] SqliteStorageEngine using SQLite version: 3.34.1
2023-04-27T19:10:45.843Z error StatsMonitor[47911] [Originator@6876 sub=StatsMonitor] Failed to execute SQL CREATE TABLE IF NOT EXISTS "sm_stats_metadata" (   stat_id         VARCHAR(128),   stat_table_name VARCHAR(128),   stat_metadata   VARCHAR(512),   CONSTRAINT "pk_sm_stats_metadata" PRIMARY KEY (stat_id),   CONSTRAINT "uniq_sm_stats_metadata" UNIQUE (stat_table_name));
2023-04-27T19:10:45.852Z error StatsMonitor[47911] [Originator@6876 sub=StatsMonitor] Unable to open and initialize storage engine: attempt to write a readonly database
2023-04-27T19:10:45.852Z info StatsMonitor[47911] [Originator@6876 sub=StatsMonitor] Exiting with status=false.

Environment

VMware vCenter Server 8.0
VMware vCenter Server 8.0.x

Cause

This issue is caused due to a race condition with 'statsmonitor' service and another service that creates entries in the system metrics DB of the vCenter system.

Resolution

This issue is resolved in vCenter Server 8.0 U1a, available at VMware Downloads.

Workaround:

To workaround the issue, please follow the below mentioned steps:

1. Perform Stage 1 deployment/upgrade process of vCenter Server.
2. Before starting Stage 2, SSH to the new VCSA and execute following commands:

touch /var/vmware/applmgmt/appliance_statsmon.sqlite
chown statsmon:statsmon /var/vmware/applmgmt/appliance_statsmon.sqlite

3. Start the Stage 2 upgrade process.