Manually Replacing an Expired VCFA 9.x Certificate Outside of VCF Operations

Products

• VCF Operations

Issue/Introduction

• The certificate for the VMware Cloud Foundation Automation (VCFA) appliance has expired.

• Automated certificate rotation or replacement via VCF Operations is failing, unavailable, or cannot be used.

• Administrator needs to manually generate and apply a new SSL/TLS certificate directly to the VCFA appliance utilizing Istio ingress.

Environment

VCF Automation 9.x 

Resolution

Below is the step-by-step instructions to manually generate a new RSA certificate chain using OpenSSL and apply it directly to the VCFA appliance Kubernetes cluster, bypassing the standard VCF Operations workflow.

Prerequisites:

• SSH access to the VCFA appliance with root privileges.
• A machine with OpenSSL installed to generate the certificates (this can be done on the VCFA appliance itself or an external Linux administrative host).
• The Fully Qualified Domain Name (FQDN) of the VCFA appliance.

Phase 1: Generate the Certificate Chain using OpenSSL

Execute the following commands on a Linux environment with OpenSSL installed. Replace <VCFA_FQDN> with the actual FQDN of your VCFA appliance.

1. Define the Common Name:

export COMMON_NAME="<VCFA_FQDN>"
# Example: export COMMON_NAME="VCFAFQDN.Domain.Com"

2. Generate the Certificate Authority (CA) Key and Certificate:

openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \ -subj "/C=US/ST=California/L=Palo Alto/O=Broadcom Inc/OU=VCF/CN=vmsp-ca"

3. Generate the TLS Key and Certificate Signing Request (CSR):

openssl genrsa -out tls.key 2048 openssl req -new -key tls.key -out tls.csr \ -subj "/C=US/ST=California/L=Palo Alto/O=Broadcom Inc/OU=VCF/CN=$COMMON_NAME" \ -config <(echo -e "[req]\ndistinguished_name=dn\nreq_extensions=req_ext\n[dn]\n[req_ext]\nsubjectAltName=DNS:$COMMON_NAME,DNS:*.$COMMON_NAME")

4. Sign the TLS Certificate with the CA:

openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt \ -days 365 -sha256 -extfile <(printf "subjectAltName=DNS:*.$COMMON_NAME")

5. Create the Full Certificate Chain:

cat tls.crt ca.crt > tls-fullchain.crt

 

Phase 2: Apply the Certificate to the VCFA Appliance

 

1. Transfer Files: If the certificates were generated on an external machine, use SCP or WinSCP to copy the tls-fullchain.crt and tls.key files to the VCFA node (e.g., to the /tmp/ directory).

2. Apply the Certificate via kubectl: SSH into the VCFA node and apply the new certificate to the Istio ingress namespace.

# Elevate to root privileges
sudo su -

# Navigate to the directory containing your certificate files
#cd /tmp/

# Export the Kubernetes Admin Config
export KUBECONFIG=/etc/kubernetes/admin.conf

# Update the Kubernetes TLS Secret
kubectl create secret tls vmsp-tls -n istio-ingress --cert=tls-fullchain.crt --key=tls.key --dry-run=client -o yaml | kubectl apply -f -

 

Phase 3: Verify the New Certificate

 

Confirm that the new certificate is being presented properly on port 443.

 

openssl s_client -connect $COMMON_NAME:443
# Alternatively: openssl s_client -connect <VCFA_FQDN>:443

Review the output to ensure the certificate dates are valid (365 days from today) and the Subject/Issuer fields match the newly generated certificate.