Resolving 'Inventory Sync Post SDDC Lifecycle Upgrade' Error in VCF Operations 9.1.x

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

 

When performing component registration, refresh, upgrade, or scale-out operations in VCF Operations, the process fails with an HTTP 400 error.

 

The following errors may be observed in the VCF Operations UI.

• An unexpected error occurred in step register_single_component_in_ops_ref__1. Reference Code: 33072210.
  Detail: Unexpected VCF Operations Suite client error has occurred. Cause: Message:

• HTTP response body: {"type":"Error","message":"Unknown error occurred.","moreInformation":[{"name":"OPERATION","value":"LOCKER_REFERENCE_CREATE"}],"httpStatusCode":400,"apiErrorCode":400}

• Refresh component in Fleet lifecycle
  Fleet lifecycle '#######' refresh has failed.

Other reference codes associated with this issue could include but not limited to: 63377924, 5507099A, and ED50381C.

This occurs even if the SDDC Manager appliance itself is healthy and its certificate is valid.

Environment

• VMware Cloud Foundation 9.1.x
• VCF Operations 9.1.x

Cause

Fleet LCM and SDDC LCM independently cache the SDDC Manager certificate once at initial pairing. This cache does not refresh automatically. When the cached certificate expires, VCF Operations rejects the SDDC Manager certificate, causing lifecycle operations to fail. The cache in the Fleet LCM namespace and the SDDC LCM namespace are independent and both may require updates.

Resolution

Workaround:

1. Identify a Control Plane Node from the VCF Operations UI: Navigate to Build > Lifecyle > VCF Management > Components > VCF Services Runtime


   
   
2. SSH into the Control Plane node with vmware-system-user (If password unknown follow steps in, "Resetting the vmware-system-user password for VMware Cloud Foundation Services Runtime")
   
   
3. Elevate to root with sudo -i

4. Run the following commands to fetch the group_id_sddc, group_id_lcm, and sddc_fqdn.

   group_id_sddc=$(kubectl exec -n vcf-sddc-lcm vcf-sddc-lcm-db-0 -c postgres -- psql -U postgres -d vcfsddclcmdb -t -A -c "SELECT sddc_group_id FROM system_info;") && echo "$group_id_sddc"
   
   sddc_fqdn=$(kubectl exec -n vcf-sddc-lcm vcf-sddc-lcm-db-0 -c postgres -- psql -U postgres -d vcfsddclcmdb -t -A -c "SELECT sddc_manager_fqdn FROM system_info;") && echo "$sddc_fqdn"
   
   group_id_fleet=$(kubectl exec -n vcf-fleet-lcm vcf-fleet-lcm-db-0 -c postgres -- psql -U postgres -d vcffleetlcmdb -t -A -c "SELECT sddc_group_id FROM sddc_lcm;") && echo "$group_id_fleet"

5. Pull the SDDC Manager certificate
   

   openssl s_client -connect $sddc_fqdn:443 -servername $sddc_fqdn -showcerts </dev/null 2>/dev/null   | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/' > $sddc_fqdn.pem

   

6. Take backups of the two config files
   

   kubectl -n vcf-fleet-lcm get configmap certificate-store -o yaml > certificate-store-vcf-fleet-lcm-backup-$(date +%Y%m%d%H%M%S).yaml
   kubectl -n vcf-sddc-lcm get configmap certificate-store -o yaml > certificate-store-vcf-sddc-lcm-backup-$(date +%Y%m%d%H%M%S).yaml

7. Build key_files
   

   KEY_NAME_fleet="entity.${group_id_fleet}.cert" && echo $KEY_NAME_fleet
   KEY_NAME_sddc="entity.${group_id_sddc}.cert" && echo $KEY_NAME_sddc

8. Create payloads
   

   PATCH_JSON_fleet=$(jq -n --arg key "$KEY_NAME_fleet" --arg cert "$(cat $sddc_fqdn.pem)" '{"data": {($key): $cert}}')
   PATCH_JSON_sddc=$(jq -n --arg key "$KEY_NAME_sddc" --arg cert "$(cat $sddc_fqdn.pem)" '{"data": {($key): $cert}}')

9. Patch the fleet and lcm config store. 
   

   kubectl -n vcf-fleet-lcm patch configmap certificate-store --type merge -p "$PATCH_JSON_fleet"
   kubectl -n vcf-sddc-lcm patch configmap certificate-store --type merge -p "$PATCH_JSON_sddc"

10. Retry the Upgrade task from the VCF Operations UI.