The rabbitmq service instance master nodes are failing

Products

• VMware Tanzu Platform

Issue/Introduction

Bosh reporting the rabbitmq service instance, 3 master nodes are in failing state: 

 

Deployment 'service-instance_xxxx-xxxx-xxxx'

Instance Process State AZ IPs VM CID VM Type Active

master/###-####-####-####  failing ap###fvc###-az1 ###.###.###.### vm-####-####-####-#### large.disk true

master/###-####-####-#### failing ap##fvc###-az1 ###.###.###.### vm-####-####-####-#### large.disk true

master/###-####-####-####  failing ap##fvc###-az1###.###.###.### vm-####-####-####-#### large.disk true

 

Attempting bosh recreate on all 3 master VMs, and they all fail to start due to the csi-controller and csi-syncer jobs are failing to run.

Environment

Rabbitmq Tile

Cause

The csi-controller job failing run on the master with connection problem to the vcenter host, and failed to complete the login due certificate issue on AD domain controllers.

csi-controller.stderr.log:

 {"level":"error","time":"2025-04-10T11:00:59.217020729Z","caller":"vsphere/virtualcenter.go:188","msg":"failed to login to vc. err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.",...

 

Resolution

Need to update the certs on the AD domain controllers to fix the failing rabbitmq service instance.