TKGI Harbor is in unresponsive state and apply change is failing with "Timed out sending 'get_state' to instance: "

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

Harbor in TKGI environment is in unresposive state and not reachable to bosh. 

When we try to apply change from the opsman UI on the Harbor tile apply change will fail with below error

Task-id: xxx, operation: update, error-message: worker/xxxxxxxxxxx: Timed out sending ‘get_state’ to instance: ‘worker/xxxxxxxxxxx’, agent-id: ‘xxxxxxxxxxxxx’ after 45 seconds

Harbor vm will be reachable to opsman vm and other jump servers ( ping the vm and test )

SSH to the Harbor VM using bosh commands will fail

 

Environment

TKGI

Cause

This might be due to the expired harbor cert or network issues

Resolution

Power off (shutdown) the Harbor VM from the vCenter and run cck command against the Harbor deployment 

 

bosh -d harbor-xxxxxxxx cck

Select option 4 - VM to recreate and wait for the process to start

If the Harbor certs are expired process will fail and then rotate the cert and apply change from the Opsman UI by selecting the Harbor tile