You may observe a new pod failing to schedule with the following error:
Warning FailedCreatePodSandBox 14s (x251 over 16h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9cck837h82jjl348934739472937293829ghc587djlaldo3l8": plugin type="nsx" failed (add): Failed to receive message header
And you will also encounter the following error in the NCP logs:
POST api/v1/logical-ports: Field level validation errors: {tags has exceeded maximum size 30}
This issue occurs when a pod is deployed with too many labels. Each label is converted to a tag on the NSX-T logical port, which has a hard limit of 30 tags per port. Additionally, NSX-T automatically creates 7 tags for each logical port when a pod is created. Consequently, the maximum number of labels a pod can have is 22.
To resolve this issue, adjust the pod deployment to include 22 or fewer labels. Once this is done, the pod should deploy successfully.