VCO Report generation fails due to Internal Error

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

VCO Report generation fails due to Internal Error. Issue seems to happen only when we include "Top Talkers" in report. 

Environment

VMware SDWAN Orchestrator

VCO version found : 6.0.0.3

Cause

Reporting Service to portal call `metrics/getEdgeFlowMetricsMultiple` is taking more than ~48 seconds to complete and the default API call timeout is set to 30seconds. 

Below are the logs seen in VCO reporting logs --->

{{{"jobId":"process.reportin~ueue.172123977.41","level":"warning","message":"Portal Invocation failed Retry times left: 0. Retrying... VcoApiClientError: VCO is unreachable\n at VcoApiClient.createApiClientError (/app/reporting/node_modules/@velocloud/service-template/build/src/lib/vcoApiClient/vcoApiClient.js:167:16)\n at VcoApiClient.callApi (/app/reporting/node_modules/@velocloud/service-template/build/src/lib/vcoApiClient/vcoApiClient.js:103:24)\n at runNextTicks (node:internal/process/task_queues:60:5)\n at listOnTimeout (node:internal/timers:540:9)\n at process.processTimers (node:internal/timers:514:7)\n at async handler (/app/reporting/build/reporting/lib/utils/retryCallVcoApi.js:33:28)\n at async retryOperation (/app/reporting/node_modules/@velocloud/service-template/build/src/lib/utils/common/common.utils.js:120:16)\n at async retryOperation (/app/reporting/node_modules/@velocloud/service-template/build/src/lib/utils/common/common.utils.js:128:16)\n at async retryOperation (/app/reporting/node_modules/@velocloud/service-template/build/src/lib/utils/common/common.utils.js:128:16)\n at async retryCallVcoApi (/app/reporting/build/reporting/lib/utils/retryCallVcoApi.js:49:22)\n at async vcoApiClient (/app/reporting/build/reporting/lib/utils/vcoApiClient.js:23:20)\n at async getEdgeFlowMetricsMultipleApi (/app/reporting/build/reporting/lib/dataAccess/enterpriseReport/metrics/edgeMetricsMultipleApi.js:15:26) {\n errorCode: 'INTERNAL_ERROR',\n httpStatus: 500,\n isUnexpected: true,\n host: '127.0.0.1',\n method: 'metrics/getEdgeFlowMetricsMultiple',\n code: 'VCO_API_ERROR',\n errors: [

{ message: 'VCO is unreachable' }
]\n}","pid":427,"timestamp":"2024-07-18T03:02:55.265Z"}}}
{{

{"jobId":"process.reportin~ueue.172123977.41","level":"error","message":"vcoApiClient call to portal threw exception: RetryOperationError: Failed to fulfill retry operation: Portal Invocation failed","pid":427,"timestamp":"2024-07-18T03:02:55.266Z"}
}}
{{

{"jobId":"process.reportin~ueue.172123977.41","level":"error","message":"vcoApiClient threw exception: RetryOperationError: Failed to fulfill retry operation: Portal Invocation failed","pid":427,"timestamp":"2024-07-18T03:02:55.266Z"}
}}
{{

{"jobId":"process.reportin~ueue.172123977.41","level":"error","message":"EdgeFlowMetricsMultipleApi.getEdgeFlowMetricsMultipleApi threw exception: RetryOperationError: Failed to fulfill retry operation: Portal Invocation failed","pid":427,"timestamp":"2024-07-18T03:02:55.266Z"}
}}

Resolution

We need to add a VCO system property as below to increase the timeout to 60 seconds. 

Name: reporting.vco.api.timeoutMs
Value: 60000
Datatype: Number

We have to use Operator Superuser to add above system property to VCO

After above change, we need to wait for sometime and then retry the report generation