VCO reporting screen doesn't load properly due to error "Unknown error occurred"
search cancel

VCO reporting screen doesn't load properly due to error "Unknown error occurred"

book

Article ID: 373919

calendar_today

Updated On:

Products

VMware SD-WAN by VeloCloud VMWare SD-WAN VMware VeloCloud SD-WAN

Issue/Introduction

VCO reporting screen doesn't load properly due to error "Unknown error occurred". We won't be able to view old reports as well as generate new reports.

When we open Monitor ---> Reports at customer level, we see below error. 

Environment

VMware SD-WAN Orchestrator
VCO version : 6.0.0.4

Cause

Reporting docker service failing to authorize due to API token issue

 

2024-08-05T15:05:00.045Z - info: [process.serviceActivation.36c7306b57c7] [1] vcoActivation Running  VCO service activation
2024-08-05T15:05:00.050Z - info: [process.fetchVco~ties.172284729.985] [7903] VCO Portal API call. Method: systemProperty/getSystemProperties. Host 127.0.0.1
2024-08-05T15:05:00.079Z - warning: [process.fetchVco~ties.172284729.985] [7903] VCO response error {"message":"Invalid token error","errorCode":"INTERNAL_ERROR","httpStatus":500,"isUnexpected":true,"name":"VcoApiClientError","host":"127.0.0.1","method":"systemProperty/getSystemProperties","code":"VCO_API_ERROR"}
2024-08-05T15:05:00.079Z - error: [process.fetchVco~ties.172284729.985] [7903] vcoApiClient call to portal threw exception: RetryOperationError: Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties
2024-08-05T15:05:00.079Z - error: [process.fetchVco~ties.172284729.985] [7903] vcoApiClient threw exception: RetryOperationError: Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties
2024-08-05T15:05:00.080Z - error: [process.fetchVco~ties.172284729.985] [7903] getVcoSystemProperties failed with error RetryOperationError: Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties
    at retryCallVcoApi (/app/reporting/build/reporting/lib/utils/retryCallVcoApi.js:52:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async vcoApiClient (/app/reporting/build/reporting/lib/utils/vcoApiClient.js:23:20)
    at async getVcoSystemProperties (/app/reporting/build/reporting/lib/jobs/fetchVcoSystemProperties.js:81:37)
    at async Object.fetchVcoSystemProperties (/app/reporting/build/reporting/lib/jobs/fetchVcoSystemProperties.js:140:35)
    at async process.handleMessage (/app/reporting/build/reporting/reporting.js:268:29) {
  errorCode: 'VCO_API_ERROR',
  httpStatus: 500,
  isUnexpected: false,
  data: {
    method: 'systemProperty/getSystemProperties',
    payload: { name: [Array] }
  },
  errors: [
    {
      message: 'Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties'
    }
  ]
}
2024-08-05T15:05:00.080Z - error: [process.fetchVco~ties.172284729.985] [7903] Job fetchVcoSystemProperties failed with error RetryOperationError: Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties
    at retryCallVcoApi (/app/reporting/build/reporting/lib/utils/retryCallVcoApi.js:52:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async vcoApiClient (/app/reporting/build/reporting/lib/utils/vcoApiClient.js:23:20)
    at async getVcoSystemProperties (/app/reporting/build/reporting/lib/jobs/fetchVcoSystemProperties.js:81:37)
    at async Object.fetchVcoSystemProperties (/app/reporting/build/reporting/lib/jobs/fetchVcoSystemProperties.js:140:35)
    at async process.handleMessage (/app/reporting/build/reporting/reporting.js:268:29) {
  errorCode: 'VCO_API_ERROR',
  httpStatus: 500,
  isUnexpected: false,
  data: {
    method: 'systemProperty/getSystemProperties',
    payload: { name: [Array] }
  },
  errors: [
    {
      message: 'Portal Invocation failed VcoApiClientError: Invalid token error. Host: 127.0.0.1. Method: systemProperty/getSystemProperties'
    }
  ]
}

 

The reporting service's token were expired.

[
  {
    "alg": "HS256",
    "typ": "JWT"
  },
  {
    "tokenUuid": "6e9faa09-aeaa-4c3a-a871-0b46c518da91",
    "exp": 1722412800,
    "uuid": "f714d263-6592-11e6-8ce3-0a16ed45d4d1",
    "iat": 1721808000
  }
]

 

The reporting server start to get error

2024-07-24T11:50:00.123Z - error: [authenticate.172180844.136775] [467858] API_TOKEN Failed to verify API token signature
2024-07-24T11:50:00.125Z - error: [authenticate.172180844.136775] [467858] error processing session ID [API_TOKEN:6e9faa09-aeaa-4c3a-a871-0b46c518da91] {"errorType":"tokenError","requestId":1721821800109,"errorMessage":"credential for authentication missing"}

 

The reporting service didn't get chance to renew the token

Resolution

Steps to fix this issue :-

 

We need to run below steps from VCO CLI using root user.

First we need to login into MySQL shell using below command and then run the steps :-

mysql --defaults-extra-file=/etc/mysql/debian.cnf velocloud



1:  Change the activation flag

update set isActivated = 0 from velocloud.VELOCLOUD_SERVICE_ACCOUNT where username = "reporting";



2: Make sure the activation key is same in the file

select * from velocloud.VELOCLOUD_SERVICE_ACCOUNT where username = "reporting";
+----+---------------------+-----------+-----------+------------------------------------------------------------------+-------------+---------------+-----------+------------------------+
| id | created             | networkId | username  | activationKey                                                    | isActivated | applicationId | urlPrefix | nginxRedirectionConfig |
+----+---------------------+-----------+-----------+------------------------------------------------------------------+-------------+---------------+-----------+------------------------+
| 12 | 2024-04-30 07:47:25 |         1 | reporting | c67a67a202f8f7442165ca5bbe5562795057d659037dc995e0bb50c3b5cfda2c |           1 |            12 | NULL      | NULL                   |
+----+---------------------+-----------+-----------+------------------------------------------------------------------+-------------+---------------+-----------+------------------------+
1 row in set (0.00 sec)

The key is stored in the file /etc/service_accounts/reporting_activation_conf.yml

 

3: Remove the expired token from reporting service

use velocloud_reporting_velocloud;
delete from VELOCLOUD_ORCHESTRATOR_INSTANCE;



4: Restart the reporting service (Below is a ubuntu command and not MySQL query so this needs to be run outside MySQL shell)

systemctl restart reporting-service


 

Powered by Wolken Software