Unified cloud proxies goes to failed Status after upgrading VMware Cloud Foundation (VCF) Operations and VMware Cloud Foundation (VCF) Operations for Logs from 9.0 to 9.0.1

Products

• VCF Operations

Issue/Introduction

 

• Attempting to configure Administration > Integration > VCF Adapter > expand VCF Instance > Edit vCenter Server Endpoint > Log Operations and enabling “Activate log collection” under Collect Logs (using the “Collect logs using the collector/group” option) completes successfully.
• However, under Infrastructure Operations > Configurations > Log Collection, an error appears stating “Failed to propagate configuration to CP.”
• Upgraded VCF Operations and VCF Operations for Logs from 9.0 to 9.0.1
• Unified Cloud Proxies Status goes to failed status with the following error in the UI displayed "Failed to propagate configuration to CP" 
• vcf-log-forwarder/log-forwarder.log in the Cloud Proxy shows error similar to :

  ERROR c.v.l.l.c.LogCollectionConfigScheduler:### - Error while fetching and applying log collection configurations.
  com.vmware.log.forwarder.httpclient.APIException: Got status 401 while retrieving session.

Environment

VMware Cloud Foundation Operations 9.0.1

VMware Cloud Foundation Operations for Logs 9.0.1

Cause

Due to corrupted credentials for internal user, the Log Forwarder cannot authenticate calls to VCF Operations for Logs, resulting in a failure of log collection.

Resolution

Broadcom is aware of this issue, and a fix will be included in the upcoming VCF Operations 9.0.2.0 release.

Workaround:

1. Take snapshot of VCF Operations for Logs appliance you can use the following documentation as a guide Take Snapshots of a Virtual Machine 
2. Open root ssh session to VCF Operations for Logs appliance primary node
3. Retrieve the session from VCF Operations for Logs in order to perform API calls.
   

   curl --request POST \
     --url https://<LI-HOST>:9543/api/v2/sessions \
     --header 'content-type: application/json' \
     --data '{
     "username": "<USERNAME>",
     "password": "<PASSWORD>",
     "provider": "Local"
   }'  -k

4. Call log-forwarder-credentials API to regenerate credentials for the Cloud Proxy Log Forwarding.
   

   curl --request PUT \
     --url https://<LI-HOST>:9543/api/v2/internal/log-forwarder-credentials \
     --header 'authorization: Bearer <LOGS-SESSION-ID>' \
     --header 'content-type: application/json' \
     --data '{
     "credentials": [
       {
         "opsProxyId": "<CLOUD_PROXY_ID>"
       }
     ]
   }' -k

5. Allow 2 to 3 collection cycles to complete and then verify that Cloud Proxies show an "Active" status under Infrastructure Operations > Configurations > Log Collection
6. If log collection is in a failed state, check the vcf-log-forwarder/log-forwarder.log in the cloud proxy for the error "Failed to create request for retrieving session."

ERROR c.v.1.1.c.LogCollectionConfigScheduler:### - Error while fetching and applying log collection configurations.
com. vmware. log. forwarder.httpclient.APIException: Failed to create request for retrieving session: null


7. If you encounter the above specified error, please proceed with the following resolution steps:
    
   1. Run the command docker ps and check that the vcf-log-forwarder container is listed
   2. Restart the vcf-log-forwarder container using the command docker restart vcf-log-forwarder.
   3. Check again using the docker ps command to confirm that the container is listed
   4. Wait for 2 to 3 collection cycles to complete.
   5. Verify that the Cloud Proxies are now showing as Active.