Missing QOE data and slow access to the VCO through the GUI

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

Edge QOE data is missing on VCO, and there is slowness in accessing the VCO portal.

QOE Data Missing:

Logs observed in VCO CLI: Backend logs

kankur@vco7-sin1:/var/log/backend$ cat velocloud.log | grep -i 'too many' | tail
2024-07-17T02:51:57.698Z - error: [process.clickhou~tion.172115247.4] [1862540] Error in cluster localhost while writing data file to clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:51:57.699Z - error: [process.clickhou~tion.172115247.4] [1862540] Error in cluster localhost while inserting data into clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:52:09.539Z - error: [process.clickhou~tion.172115664.4] [2387565] Error in cluster localhost while writing data file to clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:52:09.540Z - error: [process.clickhou~tion.172115664.4] [2387565] Error in cluster localhost while inserting data into clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:52:10.299Z - error: [process.clickhou~tion.172115916.4] [2693106] Error in cluster localhost while writing data file to clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:52:10.299Z - error: [process.clickhou~tion.172115916.4] [2693106] Error in cluster localhost while inserting data into clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))
2024-07-17T02:52:51.453Z - error: [process.clickhou~tion.172115513.4] [2205496] Error in cluster localhost while writing data file to clickhouse: Error: Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 400. (TOO_MANY_SIMULTANEOUS_QUERIES) (version 22.3.13.80 (official build))

Environment

All Vmware Velocloud Sdwan-sdwan

Resolution

We need to work with the ClickHouse team to reduce CPU usage for ClickHouse process