Generating and Downloading log bundle for VCF Automation from VCF Fleet Management UI fails

Products

VCF Operations VCF Automation

Issue/Introduction

When attempting to generate a log bundle for VCF Automation (VCFA) via VCF Fleet Management, the task appears to complete successfully, but when checked on VCF Automation appliance, the generated file is empty. But when downloading the generated log bundle it fails with the following error:

["status":"ERROR", "statusCode":"INTERNAL_SERVER_ERROR", "message":"Error occurred while downloading log bundle","resourceIdentifier":null,"errorCode":0,"errors":null}

Following is observed:

In VCF Operations > Fleet Management > Lifecycle > VCF Management > Manage VCF Automation, the VCF Automation component details; vmware-system-user password alias differs from the alias marked as "in use" in the Locker.
Triggering an Inventory Sync for VCF Automation does not resolve the alias mismatch.
The generated log file, when opened from SSH of VCF Automation appliance, contains an empty HTML or JSON string indicating a 401 Unauthorized error rather than a valid .tar.gz archive.

The /var/log/vrlcm/vmware_vrlcm.log file in VCF Fleet Management displays the following error stack trace:

INFO vrlcm[#######] [http-nio-8080-exec-6] [c.v.v.l.l.c.DeploymentsController]  -- Rest GET call to invoke the downloadurl:https://<VCFA_VIP>/supportbundles/vcfa-bundle-########T######Z-########T######Z-#####.tar.gz
...
ERROR vrlcm[#######] [http-nio-8080-exec-4] [c.v.v.l.l.c.InputValidationExceptionHandlingController]  -- Handling RuntimeException with error message: Error occurred while downloading log bundle
java.lang.RuntimeException: Error occurred while downloading log bundle
...
Caused by: java.io.IOException: Failed to download file: 401
        at com.vmware.vrealize.lcm.lcops.controller.DeploymentsController.downloadLogBundleFile(DeploymentsController.java:3052)

Environment

VCF Automation 9.x.x
VCF Fleet Management 9.x.x

Cause

This issue is caused by a dual authentication failure:

Out-of-Sync Secrets: The vmware-system-user password was changed outside of VCF Fleet Management, and the internal Kubernetes secrets on the VCFA appliance were not properly updated to reflect this new password.
Expired Root Password: The root account password on the VCFA appliance has expired. Log bundle generation requires temporary sudo privilege escalation to read Kubernetes configurations. The expired root password blocks this escalation, crashing the log generation process and resulting in an empty archive and subsequent 401 download error.

Resolution

Prerequisites:

Before proceeding with the steps below, ensure that the vmware-system-user password is known and that SSH access is available on all nodes of the VCF Automation cluster. If not, refer to the KB article below for guidance:
Unable to remediate VCF Automation "vmware-system-user" password once expired.

To resolve this issue, you must manually update the Kubernetes secret for the vmware-system-user and remove the password expiration policy for the root account.

Generate a new secret for the vmware-system-user
1. SSH into the VCFA appliance using the vmware-system-user account.
2. Run the following command to generate a new password hash (replace YOUR-NEW-PASSWORD with the correct password):
```
HASH=$(echo 'YOUR-NEW-PASSWORD' | vmsp passwd --password-stdin)
```
3. Base64 encode the hash to create the secret string:
```
SECRET=$(echo $HASH | base64 -w 0)
echo $SECRET
```
4. Copy the output of the echo $SECRET command.
Apply the updated secret to the Kubernetes Cluster
1. Elevate your privileges to root and load the Kubernetes admin config:
```
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf
```
2. Locate the exact name of the SSH password secret: (Note the output, which will look similar to vcf-mgmt-########-ssh-password-secret)
```
kubectl get secrets -n vmsp-platform | grep ssh
```
3. Patch the secret with the Base64 string you copied earlier (replace the secret name and [OUTPUT-OF-$SECRET] from Step 1(iv) with your actual values):
```
kubectl patch secret vcf-mgmt-########-ssh-password-secret -n vmsp-platform -p "{\"data\":{\"sshPassword\":\"[OUTPUT-OF-$SECRET]\"}}"
```
Fix the Root Password Expiry
1. While still running as root, run the following command to set the root password to never expire:
```
chage -m 0 -M -1 root
```
Sync and Verify
1. Log in as admin to VCF Operations UI ( https://<VCF_OPS_FQDN_IP)/ui )
2. Expand the Left Panel, Navigate to Fleet Management > Lifecycle > VCF Management > Manage VCF Automation
3. Trigger an Inventory Sync for the VCF Automation environment.
4. Verify that the Automation component vmware-system-user password alias now matches the "in use" reference in the Locker.
5. Initiate a new log bundle generation.
6. SSH to the VCF Automation appliance as vmware-system-user and run the following commands to list generated log bundle ensuring it is not a empty generated log bundle:
```
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf
K8S_TOKEN=$(kubectl get secrets synthetic-checker-krp -n vmsp-platform -ojsonpath={.data.token} | base64 -d)
PRIMARY_VIP=$(kubectl get gateway/vmsp-gateway -n istio-ingress -ojsonpath={.status.addresses[0].value})
curl -k -XPOST -H "Authorization: Bearer ${K8S_TOKEN}" "https://${PRIMARY_VIP}:30005/webhooks/vmsp-platform/supportbundle/list"
```
7. Once the task finishes, download the log bundle from the Fleet Management UI. It should now download the full, populated .tar.gz archive.

Additional Information

Run the following command on Fleet Management appliance as root user to enable the VCF Fleet Management UI to access Locker:
```
touch /var/lib/vrlcm/UI_ENABLED
```

You can monitor the live log generation progress on the VCFA appliance by running:

#Identify the pod responsible for generating the logs: (Eg: supportbundle-vmsp-platform-generate-hook-######-##### )
kubectl get pods -A

#Modify the below command with the correct pod name to view the logs
kubectl logs -f supportbundle-vmsp-platform-generate-hook-######-##### -n vmsp-platform

Following KBs can be referred: