Patching VCF Identity Broker 9.0.x fails with Error Code LCMVMSP10036

Products

• VCF Operations

Issue/Introduction

When attempting to apply a patch to VCF Identity Broker 9.0.x via VCF Operations Fleet Lifecycle UI, the task fails. The VCF Operations Fleet Lifecycle UI displays the following error workflow indicating a failure to write to the internal registry due to a certificate expiration:

Error Code: LCMVMSP10036
Applying product patch failed.
-- BUNDLE [vidb-external/vidb-########] STATUS: Failed -- remote write: Get "https://registry.vmsp-platform.svc.cluster.local:5000/v2/": tls: failed to verify certificate:x509: certificate has expired or is not yet valid: current time <date&time> is after <date&time>; GET http://registry.vmspplatform.svc.cluster.local:5000/v2/:unexpected status code 400 Bad Request: Client sent an HTTP request to an HTTPS server.

Checking the logs on the registry pod on VCF Identity Broker control plane reveals the TLS handshake rejection:

#k logs -n vmsp-platform deployment/registry

http: TLS handshake error from <IP_ADDRESS>:<PORT>: client sent an HTTP request to an HTTPS server
http: TLS handshake error from <IP_ADDRESS>:<PORT>: remote error: tls: bad certificate

Environment

• VCF Operations 9.0.x
• VCF Fleet Lifecycle 9.0.x
• VCF Identity Broker 9.0.x

Cause

The system is designed to automatically rotate the internal registry certificates. However, in some instances, the active registry pod fails to automatically restart to pick up the newly generated certificate. As a result, the pod continues serving the expired certificate, leading to a TLS handshake failure when VCF Fleet Lifecycle attempts to push the patch payload.

Resolution

To resolve the error and successfully apply the patch, manually force the registry deployment to restart, which will spawn a new pod utilizing the valid, rotated certificates. Follow the steps mentioned below:

1. Access the control plane of VCF Identity Broker node via SSH using the vmware-system-user account.
2. Elevate to root privileges by running:

   sudo -i

3. Restart the registry deployment within the platform namespace:

   kubectl rollout restart deployment/registry -n vmsp-platform

4. Wait for the new pod to initialize. You can verify the new pod is running and note its fresh startup time by executing:

   kubectl get pods -n vmsp-platform | grep registry

5. Return to VCF Operations Fleet Lifecycle UI and click Retry on the failed product patch operation. The patch bundle will now successfully upload to the internal registry.