VCF Network Insight upgrade from Fleet fails from 9.0.2 to 9.1 due to out-of-band certificate refresh

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When you attempt to upgrade VCF Network Insight (formerly known as vRealize Network Insight) from version 9.0.2 to 9.1 via the VCF Fleet Lifecycle Manager, the upgrade operation fails.
• This issue occurs after you perform an out-of-band (OOB) certificate refresh on the component, which causes a mismatch between the active component certificates and the centralized truststores.

Environment

 

• VMware Cloud Foundation (VCF) Fleet Lifecycle  9.1

• VCF Network Insight 9.0.2

 

Cause

• The out-of-band certificate refresh updates the local SSL/TLS certificate on the VCF Network Insight component but does not automatically populate those changes into the SDDC LCM truststore or the Lifecycle  truststore. Because the central tracking managers retain stale certificates, the fleet ecosystem cannot validate a secure handshake with the runtime environment, blocking the upgrade sequence.

Resolution

To resolve this issue, you must log into the VCF Services Runtime appliance to generate a VSP access token, extract the active certificate, and execute an API refresh task to sync the truststores across the Fleet Lifecycle environment.

Step 1: Generate the VSP Token and Extract Certificates

1. Log into the command line of the VCF Services Runtime appliance node.

2. Initialize the Kubernetes environment configuration context:

   cat /etc/kubernetes/admin.conf
   export KUBECONFIG=/etc/kubernetes/admin.conf
   

3. Extract and pull the newly updated certificate chain from your component by executing the following command:

   echo | openssl s_client -connect <COMPONENT-FQDN>:443 -showcerts 2>/dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > <COMPONENT-FQDN>.pem
   

4. Run the following API call to generate your authenticated VSP access token:

   export VSP_TOKEN=$(curl -ks --request POST \
     --url "https://${VSP_API_SERVER}/api/v1/identity/token" \
     --header 'Content-Type: application/x-www-form-urlencoded' \
     --data grant_type=password \
     --data "[email protected]" \
     --data "password=${VSP_PASSWORD}" \
     | jq -r '.access_token')
   

5. Verify that the token string successfully populated and is not empty or null:

   echo $VSP_TOKEN
   

Step 2: Refresh the Fleet Component Truststore

1. Review the full process context for out-of-band workflows if necessary  via the internal technical guide: Update certificate in FLEET SDDC LCM after OOB cert refresh.

2. Issue the following tracking task request to push the runtime truststore data directly into the SDDC and Lifecycle controllers:

   curl -k -X POST --url "https://${INSTANCE_FQDN}/sddc-lcm/v1/components/{componentId}?action=refresh" \
     -H "Authorization: Bearer $VSP_TOKEN" \
     -H "Content-Type: application/json"
   

3. Retrying the upgrade sequence through the Fleet Lifecycle dashboard should now proceed without encountering trust errors.