kubectl commands fails in pinniped integrated clusters with error "Unable to connect to the server: An existing connection was forcibly closed by the remote host"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Requests using the pinniped workflow to fetch kubernetes objects were failing with below error in pinniped pod logs

{"level":"info","timestamp":"2024-07-25T15:34:05.594448Z","caller":"k8s.io/[email protected]/trace/trace.go:236$trace.(*Trace).logTrace","message":"Trace[1300955003]: \"Create\" accept:application/json,audit-id:e97ac68c-aba5-4564-8700-ac6666b528e8,client:10.32.210.16,protocol:HTTP/2.0,resource:tokencredentialrequests,scope:resource,url:/apis/login.concierge.pinniped.dev/v1alpha1/tokencredentialrequests,user-agent:tanzu-pinniped-v0.23.0-client-v0.32.2-a948c6586/v0.0.0 (linux/amd64) kubernetes/$Format,verb:POST (25-Jul-2024 15:33:05.593) (total time: 60001ms):\nTrace[1300955003]: ---\"limitedReadBody failed\" len:0,err:stream error: stream ID 213; CANCEL 59997ms (15:34:05.590)\nTrace[1300955003]: [1m0.001330642s] [1m0.001330642s] END\n"}

Cause

• The TokenCredentialRequest was failing to complete in 60s because the size of sessions.yaml and credentials.yaml was more due to large number of groups returned by Pinniped.
• File size should be less then 10KB ideally. 
• An mTLS client cert can have approximately 60k total characters (roughly, not the exact limit). This is because golang TLS handshakes are limited to size 65,536. So if you look at the ID token in sessions.yaml and add up the total number of characters and if total is more than 60k then it will fail.
• If search filter is configured to "filter: (objectClass=group)" means every user should be treated as if they belong to every single LDAP group in the whole LDAP database. This can increase the size of sessions.yaml file.
  
  

Resolution

To decrease the number of groups returned by the LDAP server, group filter parameter can be set to &(sAMAccountName={})(objectclass=group) to filter out groups where only particular user was a member of.