Recovering from mTLS issues after upgrading from PAIS 2.0.*

Products

VCF Private AI Services

Issue/Introduction

After upgrading a Private AI Services (PAIS) deployment from PAIS 2.0.x to PAIS 2.1.0 or later, you may notice degraded or intermittently unavailable Agent Builder and/or Data Indexing and Retrieval (REX) functionality some time after the upgrade completes - typically weeks to a few months later, not immediately.

When inspecting the affected tenant namespace, you will typically observe:

Repeated restarts of the rex-worker pod, and potentially other pods that participate in PAIS's internal mTLS network (for example rex-mcp-server, mcp-monitor).
Logs for these pods show TLS/mTLS handshake or certificate verification errors when calling other PAIS internal services. The underlying error at system level would usually appear as a log entry containing the message:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate_verify_failed: certificate has expired (_ssl.c:1016)
```

Environment

Private AI Services (PAIS) - A tenant namespace originally installed on PAIS 2.0.x and later upgraded to PAIS 2.1.0 or later. A tenant installed directly on PAIS 2.1.0 or later (without going through this upgrade path) is not affected.
VMware Cloud Foundation (VCF): 9.0 or 9.1 with Private AI Services installed.

Cause

PAIS provides several internal, cluster-private certificates via cert-manager. Specifically, a root CA, an intermediate CA used for internal mTLS, and a leaf certificate (signed by that intermediate CA) used by PAIS components to authenticate to each other.
On PAIS 2.0.x, the root and intermediate CA certificates were issued without an explicit lifetime, so they defaulted to cert-manager's built-in 90-day lifetime.

Starting with PAIS 2.1.0, PAIS requests a much longer lifetime for these CA certificates. However, a tenant that upgraded from 2.0.x (rather than installing fresh on 2.1.0+) keeps renewing the intermediate CA on its original 90-day schedule for one additional cycle after the upgrade, rather than adopting the new lifetime immediately.
Because the intermediate CA and the mTLS leaf certificate it signs renew independently of each other, this can leave the two out of step, which has been observed to cause the intermittent internal mTLS validation failures described above.

Resolution

Note:

Run the below steps against the affected PAIS tenant namespace (where the PAISConfiguration and the failing pods live) - not the PAIS Supervisor Service namespace.
This procedure only recreates two internal certificates. It does not touch application data, the PAISConfiguration resource, or any customer-facing certificate (such as the ingress certificate), and it does not require reinstalling PAIS.

To resolve the issue, delete the intermediate CA secret and the mTLS leaf secret. This forces cert-manager to reissue both certificates from scratch, back in step with each other, which resolves the mismatch.

Set up kubectl access to the affected tenant's namespace, if you haven't already:
```
kubectl vsphere login --server=<SUPERVISOR-CONTROL-PLANE-IP-ADDRESS> --vsphere-username <VCENTER-SSO-USER>
kubectl config get-contexts
kubectl config use-context <tenant-namespace-context>
```
This requires the vSphere Plugin for kubectl to be installed and the Supervisor's control plane certificate to be trusted on your system.
See, Get and Use the Supervisor Context for full details, including how to find your Supervisor's control plane IP address.
Confirm you're pointed at the correct tenant namespace and locate the affected resources.

Every PAIS-managed certificate and secret carries a pais.vmware.com/component label, so you don't need to know their exact (UID-suffixed) names:
```
kubectl config view --minify -o jsonpath='{..namespace}'
kubectl get certificate,secret -l 'pais.vmware.com/component in (mtls-ca,mtls)'
```
You should see one Certificate and one Secret for each of mtls-ca (the intermediate CA) and mtls (the leaf certificate).
Delete the intermediate CA secret first, and wait for cert-manager to reissue it:
Note: There is no need to back up this secret, as it is a managed resource and will be automatically re-created if missing.
```
kubectl delete secret -l 'pais.vmware.com/component=mtls-ca'
kubectl wait --for=condition=Ready certificate -l 'pais.vmware.com/component=mtls-ca' --timeout=2m
```

Once the intermediate CA has been reissued, delete the mTLS leaf secret so it is reissued using the new intermediate CA, and wait for it to complete:

kubectl delete secret -l 'pais.vmware.com/component=mtls'
kubectl wait --for=condition=Ready certificate -l 'pais.vmware.com/component=mtls' --timeout=2m

Confirm the affected pods stop restarting and no longer log mTLS errors:
```
kubectl get pods -l 'pais.vmware.com/component in (rex-worker,rex-mcp-server,mcp-monitor,api)' --watch
kubectl logs -l 'pais.vmware.com/component=rex-worker' --tail=50
```
PAIS's cert-reloader mechanism picks up the refreshed certificates automatically, so manual pod restarts should not be required. If errors persist after a few minutes, restart the affected Deployment as a final step, for example:
```
kubectl rollout restart deployment -l 'pais.vmware.com/component=rex-worker
```

Additional Information

If the symptoms persist after completing these steps, or you are unsure whether they match your situation, collect a PAIS support bundle and contact Broadcom Support.