TKGI CLI commands show error message "od-broker is processing a request for the same instance"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Running certain TKGI CLI commands return the following error:

Error: Status: 500; ErrorMessage: <nil>; Description: od-broker is processing a request for the same instance: <VM instance UUID>, please try again later; ResponseError: <nil>

Example commands that could report the above error:
 

• pks get-kubeconfig CLUSTERNAME
• pks get-credentials CLUSTERNAME

The pks-api.log shows similar errors:

ERROR 15625 --- [nio-9021-exec-2] i.pivotal.pks.error.ApiExceptionHandler  : Error: Status: 500; ErrorMessage: <nil>; Description: od-broker is processing a request for the same instance: fea7b3f8-b5b9-4ee4-843f-0c0c29689664, please try again later; ResponseError: <nil>

Environment

Product Version: 1.9

Resolution

These errors can occur if there are a large number of requests coming in, and a request is not able to be processed within 10 seconds to the On Demand Broker (od-broker).

Starting with the following releases, this issue has been addressed by TKGI Engineering:
 

• TKGI 1.9.5
• TKGI 1.10.1
• TKGI 1.11

To address this issue, please upgrade to these releases or higher.

Note: If the issue is persistent instead of intermittent, the workaround is to restart the broker service on the pivotal-container-service VM. 

bosh ssh -d <pivotal-container-service-ID> pivotal-container-service -c "sudo monit restart broker"