Opsman Deployments Failing with Monit Error: "monit-actual: Cannot connect to the monit daemon. Did you start it with http support?"

Products

• Operations Manager

Issue/Introduction

Applying changes in Opsman fails with a monit error:

Task 2125172 | 01:11:46 | Updating instance diego_brain: diego_brain/242c1d84-########-f42619dbbe1e (1)
Task 2125172 | 01:11:54 | L executing pre-stop: diego_brain/242c1d84-#########-f42619dbbe1e (1)
Task 2125172 | 01:11:54 | L executing drain: diego_brain/242c1d84-#########-f42619dbbe1e (1)
Task 2125172 | 01:12:10 | L stopping jobs: diego_brain/242c1d84-##########-f42619dbbe1e (1) (00:00:30)
                        L Error: Action Failed get_task: Task 0eb0a3e5-##########-1555e1a9c866 result: Stopping Monitored Services: Stop all services: Running command: 'monit stop -g vcap', stdout: '', stderr: 'monit-actual: Cannot connect to the monit daemon. Did you start it with http support?
': exit status 1

/var/vcap/monit/monit.log shows a similar error:

[UTC Jul  7 17:03:29] info     : stop service 'otel-collector' on user request
[UTC Jul  7 17:03:29] info     : monit-actual daemon at 12860 awakened
[UTC Jul  7 17:03:29] info     : Awakened by User defined signal 1
[UTC Jul  7 17:03:29] info     : 'otel-collector' stop action done
[UTC Jul  7 17:03:34] error    : monit-actual: Cannot connect to the monit daemon. Did you start it with http support?

This issue may happen intermittently on random BOSH VMs

Environment

Operations Manager / Foundation Core

Ubuntu Jammy Stemcell

Cause

The iptable rules for bosh-agent to access monit daemon are inserted in an improper order, it could drop some packets in certain edge cases, therefore BOSH deployment may encounter the error during the phase of instance drain intermittently. 

Resolution

This issue is fixed in Jammy Stemcell v1.1329 and above. Please upgrade stemcell to fixed versions. 

If the stemcell in your foundations are already at the impacted versions v1.1193 ~ v1.1328, the deployment will possibly encounter the issue, please work around it by steps:

1. Recreate Failing Instance:

   `bosh recreate <instance-id> --no-converge` on the failing VM.

2. Resume the Apply Changes. The deployment will resume from the failing point.