Error : "Transfer Failed Due To An Error. Please try again" Log Data Transfer Fails

Products

• VCF Operations

Issue/Introduction

• In VCF Operations, navigating to Administration > Control Panel > Log Data Transfer and clicking the "Transfer Logs" button fails with the following error message on the UI:
  

  Transfer failed due to an error. Please try again

• When reviewing the /storage/log/vcops/log/vcfops-bridge.log log on the VCF Operations side, connection timeouts to the VCF Operations for Logs appliance. The bridge sent a POST /api/v2/data-migrations to the Logs appliance and never received a response within the read timeout. After 3 retries it gave up:
  

  WARN  vcfops-bridge ... LogInsightRestTemplate.doExecute
    - Request POST https://<FQDN-VCF-Ops-Logs-node>:9543/api/v2/data-migrations
      encounter error: I/O error on POST request ...: Read timed out;
      nested exception is java.net.SocketTimeoutException: Read timed out.
      Remain retry count: 3
  WARN  vcfops-bridge ... Remain retry count: 2
  WARN  vcfops-bridge ... Remain retry count: 1
  ERROR vcfops-bridge ... OpsLogsManager.exchangeInternal
    - java.net.SocketTimeoutException: Read timed out
  ERROR vcfops-bridge ... BaseServerManager.logAndReportError
    - Logs could not be retrieved as the connection timed out.
  ... Function runLIDataTransfer execution takes 243322 
  Total time spent waiting: ~243 seconds. 

Environment

• VCF Operations 9.0.2
• VCF Operations for Logs 9.0.2

Cause

The primary cause of this issue is a network firewall configuration blocking Port 22 (SSH) between the VCF Operations for Logs nodes. During the log transfer initialization, the DataMigrationController attempts to contact all source nodes via SSH. If Port 22 is blocked, the connection times out, which cascades into the SocketTimeoutException seen in the VCF Operations UI.

This is confirmed in the /storage/core/loginsight/var/runtime.log of the VCF Operations for Logs appliance:

[DataMigrationController]
[startMigration method called with request payload:
DataMigrationRequest{username='root', password='{24}',
                        hostName='####',
                        bucketStartTimeMillis=1776086201836,
                        bucketEndTimeMillis=1776950201836}] 
This is where it failed (and continued failing on every subsequent retry):
[globals.GlobalErrorHandler]
  [Error executing request POST /api/v2/data-migrations]
java.util.concurrent.CompletionException:
  com.vmware.loginsight.migration.DataMigrationException:
    ssh: connect to host #.#.#.# port 22: Connection timed out
    at com.vmware.loginsight.api.providers.migration.DataMigrationProvider.startMigration(...)
ERROR ... Error executing request POST /api/v2/data-migrations
   ...ssh: connect to host #### port 22: Connection timed out

Resolution

1. Verify network connectivity and ensure that Port 22 (TCP) is open between all VCF Operations for Logs nodes.

2. Update any intervening network firewalls or security groups to permit SSH traffic across the logging cluster nodes.

3. Once Port 22 is confirmed open, return to the VCF Operations administration UI and retry the Log Data Transfer operation.