Error: Timed out sending 'fetch_logs_with_signed_url' to instance

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

A Windows Diego Cell appears as unresponsive agent in bosh vms. Attempts to retrieve logs fail with a timeout error.

Symptoms:

• Command bosh logs <INSTANCE> fails.
• Error: Timed out sending 'fetch_logs_with_signed_url' to instance: 'windows_diego_cell/####', agent-id: '####' after 45 seconds.
• The instance status in bosh vms is unresponsive agent.

Environment

• VMware Tanzu Application Service (TAS) for VMs [Windows]
• Windows Diego Cell

Cause

The BOSH Director is unable to communicate with the BOSH Agent running on the Windows VM. This can be caused by the agent process hanging, a duplicate IP on the network, or the VM being in a hung state at the IaaS layer. Because the communication is broken, the Director cannot initiate the "drain" process or collect logs.

Resolution

To recover the unresponsive cell:

1. Identify the deployment and instance ID.
2. Run the recreate command with the --fix flag to force the replacement of the unresponsive VM: bosh -d <DEPLOYMENT> recreate <INSTANCE_GROUP>/<INDEX> --no-converge --fix
   • The --no-converge flag instructs the BOSH Director to restrict the action to only the specified VM/instance, whereas it ignores the deployment manifest's intended state and instead recreates the instance based exactly on the current state recorded in the database.
   • The --fix flag instructs the BOSH Director to replace the VM even if the current agent is not responding.
3. Verify the health of the instance afterwards.  If the issue persists, verify there are no duplicate IPs in the environment for the assigned IP address.

Additional Information

For more details on managing unresponsive agents, see the KB: Bosh VMs are fluctuating between 'Unresponsive Agents' and Healthy states (297508).