No organizations are imported after upgrade from VRA 8 to VCFA 9.0

Products

• VCF Automation

Issue/Introduction

After upgrading from VRA 8 to VCFA 9.0, users can log in only to the System organization and no additional organizations are available

The issue can be identified in the log bundle under /services-logs/prelude/<node>/vmsp-prelude-deployer-app/deploy/

SSLError(SSL│CertVerificationError("Certificate fingerprints don't match.
(old thumbprint) <> (new thumbprint)))

Environment

VCF Automation 9.0.x

Cause

VCFA failed to verify the VMware Identity Manager (vIDM) SSL certificate because the certificate fingerprint presented by vIDM did not match the expected fingerprint during upgrade validation. Due to this SSL certificate mismatch, identity data migration was skipped and tenant organizations were not created.

Resolution

Apply the following workaround (no need to revert):

SSH to a VCFA Node and do the following:

k get vaconfigs.prelude.vmware.com prelude-vaconfig -o=jsonpath="{.spec.vidm}" |jq -r .url #get the hostname only

#Retrieve leaf certificate:
VIDM_HOST=<vidmhostname>
CERT_PEM=`openssl s_client -connect $VIDM_HOST:443 </dev/null 2>/dev/null | openssl x509 -outform PEM`
# if vidm is accessible only via proxy - use this instead of the line above:
#CERT_PEM=`k exec -it -n prelude identity-service-app-<...> -- openssl s_client -connect $VIDM_HOST:443 </dev/null 2>/dev/null | openssl x509 -outform PEM`

#Update configuration used by upgrade
SPEC=`jq -nc --arg cert "${CERT_PEM}" '{spec: {vidm: {cert: $cert}}}'`
kubectl patch vaconfig prelude-vaconfig --type='merge' -p "${SPEC}"

#Re-run migration
k8s_token=$(kubectl get secrets synthetic-checker-krp -n vmsp-platform -ojsonpath=\{.data.token} | base64 -d)
curl -k -X POST https://localhost:30005/webhooks/prelude/data-migration/run -H "Authorization: Bearer $k8s_token"