Apply changes fails on “update-admin-password” errand for Healthwatch tile.
search cancel

Apply changes fails on “update-admin-password” errand for Healthwatch tile.

book

Article ID: 409889

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

  • After an upgrade of Healthwatch tile from v2.3.2 to v2.3.3 , "Apply Changes" fails on “update-admin-password” errand. 
  • You will observe below mentioned error in the bosh task log. 
Using deployment 'p-healthwatch2-2f9eXXXXXX'

Task 16559184

Task 16559184 | 02:25:31 | Preparing deployment: Preparing deployment
Task 16559184 | 02:25:31 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 16559184 | 02:25:31 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 16559184 | 02:25:31 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 16559184 | 02:25:31 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 16559184 | 02:25:32 | Preparing deployment: Preparing deployment (00:00:01)
Task 16559184 | 02:25:32 | Running errand: grafana/80c12bbXXXXXX(1) (00:00:51)
Task 16559184 | 02:26:23 | Fetching logs for grafana/80c12bbXXXXXXXX (1): Finding and packing log files (00:00:01)

Task 16559184 Started  Fri Aug  8 02:25:31 UTC 2025
Task 16559184 Finished Fri Aug  8 02:26:24 UTC 2025
Task 16559184 Duration 00:00:53
Task 16559184 done
Errand 'update-admin-password' completed with error (exit code 1)

Instance   grafana/80c12bbXXXXXXXX
Exit Code  1
Stdout     Deprecation warning: The standalone 'grafana-cli' program is deprecated and will be removed in the future. Please update all uses of 'grafana-cli' to 'grafana cli'
           logger=settings t=2025-08-08T02:25:32.689595842Z level=info msg="Starting Grafana" version= commit= branch= compiled=1970-01-01T00:00:00Z
           logger=settings t=2025-08-08T02:25:32.69012999Z level=info msg="Config loaded from" file=/var/vcap/packages/grafana/conf/defaults.ini
           logger=settings t=2025-08-08T02:25:32.690726887Z level=info msg="Config loaded from" file=/var/vcap/jobs/grafana/config/grafana.ini
           logger=settings t=2025-08-08T02:25:32.690869263Z level=info msg=Target target=[all]
           logger=settings t=2025-08-08T02:25:32.690961142Z level=info msg="Path Home" path=/var/vcap/packages/grafana
           logger=settings t=2025-08-08T02:25:32.691059542Z level=info msg="Path Data" path=/var/vcap/store/grafana
           logger=settings t=2025-08-08T02:25:32.691138935Z level=info msg="Path Logs" path=/var/vcap/sys/log/grafana
           logger=settings t=2025-08-08T02:25:32.69122122Z level=info msg="Path Plugins" path=/var/vcap/store/grafana/grafana_plugins
           logger=settings t=2025-08-08T02:25:32.691302212Z level=info msg="Path Provisioning" path=/var/vcap/jobs/grafana/config/provisioning
           logger=settings t=2025-08-08T02:25:32.69137982Z level=info msg="App mode production"
           logger=sqlstore t=2025-08-08T02:25:32.691618328Z level=info msg="Connecting to DB" dbtype=mysql
           logger=migrator t=2025-08-08T02:25:32.710338174Z level=info msg="Starting DB migrations"
           logger=migrator t=2025-08-08T02:25:32.720222108Z level=info msg="migrations completed" performed=0 skipped=526 duration=717.643µs
           logger=secrets t=2025-08-08T02:25:32.728960723Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
Exit code 1
           Error: ✗ failed to update user password: Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Stderr     Error: failed to run job-process: exit status 1 (exit status 1)


1 errand(s)

 

Environment

Seen in Healthwatch 2.3.3 (it may happen with other versions too)

Cause

This error is related to MySQL/Galera and it basically means that a database transaction waited too long for a lock and was aborted.

During an "Apply Changes" of a TAS (or Healthwatch) tile, Ops Manager updates credentials in the internal MySQL database. If another process is holding a lock on that table or row, MySQL waits for the lock. If the wait exceeds the configured innodb_lock_wait_timeout (often 50s or 120s), MySQL returns:

Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

 

Resolution

To check the current locks and transactions at the time of the issue:

1. SSH to the MySQL VM (for health watch deployment) via BOSH:

bosh  -d p-healthwatch2-xxxxxx ssh pxc/xxxxx

 

2. Connect to MySQL:

pxc/xxxxx:~$ sudo -i 

pxc/xxxxx:~# mysql -u admin -p

 

3. Run below command to check what MySQL server is doing right now and Identify which query is waiting for a lock.:

SHOW PROCESSLIST;

For example :

+---------+-----------------------+--------------------+---------+---------+---------+--------------------------+------------------+------------+-----------+---------------+
| Id      | User                  | Host               | db      | Command | Time    | State                    | Info             | Time_ms    | Rows_sent | Rows_examined |
+---------+-----------------------+--------------------+---------+---------+---------+--------------------------+------------------+------------+-----------+---------------+
|       1 | system user           |                    | NULL    | Sleep   | 4226634 | innobase_commit_low (-1) | NULL             | 4226634376 |         0 |             0 |
|       2 | system user           |                    | NULL    | Sleep   | 4226634 | wsrep aborter idle       | NULL             | 4226634376 |         0 |             0 |
|      12 | cluster-health-logger | localhost          | NULL    | Sleep   |      16 |                          | NULL             |      16356 |         1 |             2 |
| 284XXXX | galera-agent          | localhost          | NULL    | Sleep   |       0 |                          | NULL             |        101 |         1 |             3 |
| 296XXXX | grafana               | 192.XXXX:57XXX | grafana | Sleep   |       8 |                          | NULL             |       8248 |         0 |             0 |
| 2969XXXX | admin                 | localhost          | NULL    | Query   |       0 | init                     | show processlist |          4 |         0 |             0 |
| 2969XXXX | grafana               | 192.XXXX:42XXX | grafana | Sleep   |       1 |                          | NULL             |        567 |         0 |             0 |
+---------+-----------------------+--------------------+---------+---------+---------+--------------------------+------------------+------------+-----------+---------------+
7 rows in set (0.00 sec)

 

4. Find the process on which lock is stuck

ps -ef | grep -i mysql

 

5. Once you confirm the process is holding the lock , ssh to pxc/XXXXXX vm and restart the processes.

monit restart all

 

Additional Information

Please note that the Errand “Update Grafana Admin Password” is optional as it updates the administrator password for the Grafana UI.

Reference Docs - https://techdocs.broadcom.com/us/en/vmware-tanzu/platform-services/healthwatch-for-vmware-tanzu/2-3/healthwatch/configuring-configuring-healthwatch.html

Powered by Wolken Software