Error: "cloud director endpoint" when browsing a key provider on VMware Cloud Director Encryption Management

Products

• VMware Cloud Director

Issue/Introduction

• When browsing a key provider on VMware Cloud Director Encryption Management on VMware Cloud Director provider portal, you see the error below:
  
  
  cloud director endpoint https://cloud.example.com:443/cloudapi/1.0.0/entities/types/vmware/byok_org_key_provider/1.0.0?filter=%28entity.keyProviderId%3D%3Durn%3Avcloud%3Aentity%3Avmware%3Abyok_key_provider%########-####-####-########34d5%29&page=1&pageSize=128 failed at byok/client/vcd.(*Client).request(/opt/src/byok/client/vcd/client.go:271) at byok/client/vcd.(*EntityRequest[...]).queryPage(/opt/src/byok/client/vcd/client_entity.go:509) at byok/client/vcd.(*Request).Get(/opt/src/byok/client/vcd/client_request.go:177) at byok/client/vcd.(*EntityRequest[...]).Query(/opt/src/byok/client/vcd/client_entity.go:145) at byok/client/vcd.QueryRuntimeEntities[...](/opt/src/byok/client/vcd/rde_entity.go:137) at byok/repository/rde.(*rdeRepository[...]).getPage(/opt/src/byok/repository/rde/repository.go:180) at byok/repository/rde.(*rdeRepository[...]).ListAll(/opt/src/byok/repository/rde/repository.go:69) at byok/controller/reconciler.(*keyProviderReconciler).collectActualOrgKeyProviders(/opt/src/byok/controller/reconciler/key_provider_reconciler.go:726) at byok/controller/reconciler.(*keyProviderReconciler).syncTenants(/opt/src/byok/controller/reconciler/key_provider_reconciler.go:474) at byok/controller/reconciler.(*keyProviderReconciler).realizePresentState(/opt/src/byok/controller/reconciler/key_provider_reconciler.go:234) at byok/controller/reconciler.(*keyProviderReconciler).realize(/opt/src/byok/controller/reconciler/key_provider_reconciler.go:205) at byok/controller/reconciler.(*keyProviderReconciler).Reconcile(/opt/src/byok/controller/reconciler/key_provider_reconciler.go:112) at byok/controller.(*Controller).reconcileHandler(/opt/src/byok/controller/controller.go:140) at byok/controller.(*Controller).processNextWorkItem(/opt/src/byok/controller/controller.go:127) at byok/controller.(*Controller).Start.func3.1(/opt/src/byok/controller/controller.go:104) at byok/util/wait.JitterUntilWithContext.func1(/opt/src/byok/util/wait/backoff.go:259) at byok/util/wait.BackoffUntil.func1(/opt/src/byok/util/wait/backoff.go:227) at byok/util/wait.BackoffUntil(/opt/src/byok/util/wait/backoff.go:211) at byok/util/wait.JitterUntil(/opt/src/byok/util/wait/backoff.go:205) at byok/util/wait.JitterUntilWithContext(/opt/src/byok/util/wait/backoff.go:260) at byok/controller.(*Controller).Start.func3(/opt/src/byok/controller/controller.go:107) at byok/controller.(*Controller).Start.func3(/opt/src/byok/controller/controller.go:103) at runtime.goexit(/usr/local/go/src/runtime/asm_amd64.s:1651): 500 Server Error(500)"

• When running the below API using Postman or similar against a specific VMware Cloud Director cell, bypassing the LB we have the same error "500 Server":
  
  https://cloud.example.com:443/cloudapi/1.0.0/entities/types/vmware/byok_org_key_provider/1.0.0?filter=%28entity.keyProviderId%3D%3Durn%3Avcloud%3Aentity%3Avmware%3Abyok_key_provider%########-####-####-########34d5%29&page=1&pageSize=128
  
  

Environment

VMware Cloud Director 10.x
 VMware Cloud Director Encryption Management 1.1.0

Cause

 The http server is limiting the http headers size. 

Resolution

Execute the following command on all the VMware Cloud Director cells to  increases the headers limit:

cmt manage-config -n "vcloud.http.responseHeaderMax" -v 102400

After the command above is executed a restart of the services on the cell is needed.