Log Collection Configuration Failure: Internal Error on vCenter Server - VCF Operations 9.1

Products

• VCF Operations
• VMware vCenter Server

Issue/Introduction

• After upgrading VCF Operations for Logs from version 9.0.2 to 9.1, activating log collection for the workload domain or management domain vCenter may fail.
• The vCenter log collection enters a failed state, and the UI displays the following tooltip error message: 

  "Failed to configure VC to send logs: Internal Error - As error occurred on the server side. The Collector attempts to configure log collection every 5 minutes."

• The management domain vCenter continues to function normally, and the VCF adapter itself remains in a collecting state with no general data collection issues.
• The following entries recorded in /storage/log/vcops/log/collector.log when attempting to activate log collection is observed: 

  INFO collector [ops@#### threadid ="####" threadName="itask-thread-log-enabled-vc-retriever"] [com.integrien.alive.common.communication. TasksManager.sendTask] - Sending task. ID: ##, Name: Log enabled VCs, CollectorID: #, Details: n/a. 
  INFO collector [ops@#### threadid ="##" threadName="TasksManager-Received Tasks Processor-#"] [com.integrien.alive.common.communication. TasksManager.run) - Received response. ID: ##, Name: Log enabled VCs, CollectorID: #, Created at: #############, Details: n/a, Elapsed Time: #### msecs.

• The following errors logged indicating authentication failures /var/log/loginsight-agent/liagent_api.log: 

  ERROR liagent_api #### [vc@####] [liagent/liagent_helper.go:liagent.manage EnvoyConfig:###] failed to get the HOK signer 
  ERROR liagent_api #### [vc@#### ] [liagent/liagent_helper.go: liagent.manageEnvoyConfig:###] failed to get the HOK signer 
  ERROR liagent_api #### [vc@####] [liagent/liagent_helper.go:liagent.cleanup:##] failed to delete the liagent envoy listener: nse: FailedAuthentication: Invalid credentials

Environment

• VCF Operations (Log Management) 9.1.x
• vCenter 9.1.x

Cause

The log collection configuration fails because the agent configuration passes invalid credentials. These repeated invalid credential attempts eventually cause the vmware-applmgmtservice account to become locked out due to multiple tries.

Resolution

1. Clear the service account's locked state
   
   • This can be accomplished by restarting the VMware Appliance Management Service (applmgmt) directly from the vCenter VAMI page by following the steps in KB Scheduled File-Based Backup fails to start on vCenter server.
   • Alternatively, performing a full reboot of the vCenter appliance will also clear the locked state.
2.  Deactivate and activate log collection on each vCenter adapter instance
   1. Access the Console
      1. Log into the VCF Operations UI. 
   2. Navigate to Integrations
      1. Go to Operate in the main menu.
      2. Click on Integrations.
   3. Disable Log Collection
      1. Select the target vCenter Adapter instance.
      2. Click Edit to open its configuration.
      3. Expand the Log Operations section.
      4. Uncheck Activate Log Collection.
      5. Click Save.
   4. Enable Log Collection
      1. Select the same vCenter Adapter instance again.
      2. Click Edit to modify the configuration.
      3. Expand the Log Operations section.
      4. Check Activate Log Collection.
      5. Click Save

Once the service has been restarted or the appliance rebooted and the Log Collection as being restarted, the log collection status should automatically update to "Collecting" in VCF Operations.