Error: 'mysql_monitor/xyz' is not running after update. Review logs for failed jobs: replication-canary

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

It may happen, that when trying to upgrade the TAS Tile, it fails with the below error.

Error: 'mysql_monitor/21234dbb-4878-47d1-bc3f-860799abcd (0)' is not running after update. Review logs for failed jobs: replication-canary

Once you check the replication-canary folder in the mysql_monitor VM you will see following error in canary.stderr.log

goroutine 1 [running]: code.cloudfoundry.org/lager.(*logger).Fatal(0xc00004e420, 0x7f4222, 0x1d, 0x870960, 0xc00048b1d0, 0x0, 0x0, 0x0) /var/vcap/data/compile/replication-canary/replication-canary/vendor/code.cloudfoundry.org/lager/logger.go:162 +0x582 main.main() /var/vcap/data/compile/replication-canary/replication-canary/main.go:99 +0x1682 panic: Post "https://uaa.sys.f1.abc.def.local/oauth/token": x509: certificate has expired or is not yet valid: current time 2022-02-15T05:42:27Z is after 2022-01-12T17:26:15Z goroutine 1 [running]: code.cloudfoundry.org/lager.(*logger).Fatal(0xc00004e420, 0x7f4222, 0x1d, 0x870960, 0xc000102bd0, 0x0, 0x0, 0x0) /var/vcap/data/compile/replication-canary/replication-canary/vendor/code.cloudfoundry.org/lager/logger.go:162 +0x582 main.main() /var/vcap/data/compile/replication-canary/replication-canary/main.go:99 +0x1682 panic: Post "https://uaa.sys.f1.abc.def.local/oauth/token": x509: certificate has expired or is not yet valid: current time 2022-02-15T05:42:47Z is after 2022-01-12T17:26:15Z goroutine 1 [running]: code.cloudfoundry.org/lager.(*logger).Fatal(0xc00004e420, 0x7f4222, 0x1d, 0x870960, 0xc0003871d0, 0x0, 0x0, 0x0) /var/vcap/data/compile/replication-canary/replication-canary/vendor/code.cloudfoundry.org/lager/logger.go:162 +0x582 main.main() /var/vcap/data/compile/replication-canary/replication-canary/main.go:99 +0x1682

The key message here from the above log is

panic: Post "https://uaa.abc.def.local/oauth/token": x509: certificate has expired or is not yet valid

Environment

Product Version: 2.11

Resolution

Check for any expired cert in your foundation and rotate it.

If you don't find any expired cert, and you have a load balancer in front of the routers, then it means that the load balancer cert is expired.