CPU Utilization is critical on Portal Prod Server. Server is currently using more than 80% CPU. docker stats is showing the portal_enterprise is consuming most of the CPU. Need your help here: -
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
5c2b1f5c4dff portal_portaldb.1.sr3mz11rluwavalm6z4mofoao 6.04% 182MiB / 62.76GiB 0.28% 8.21GB / 273GB 4.06MB / 101GB 92
669519a12a8d portal_middlemanager.1.w9bsb03rp5ffu466o7jk9tyic 1.06% 2.643GiB / 4GiB 66.07% 4.29TB / 35.7GB 467MB / 1.69GB 385
9df28f80579c portal_ingestion-server.2.7aspfo213d8xvnidlkpsige2k 0.07% 655.9MiB / 1.5GiB 42.70% 251GB / 1.61TB 0B / 0B 51
c8a8d158c526 portal_analytics-server.44f3a66ccxvnntntfchzjttfq.2i65uvppvpyc9tkgm0ptv8r00 0.06% 431MiB / 4.4GiB 9.56% 313MB / 216MB 1.88MB / 0B 83
6bc663bb1d57 portal_dispatcher.44f3a66ccxvnntntfchzjttfq.ikzu77ipu7cngdp7tlqyly22l 0.06% 22.99MiB / 1.099GiB 2.04% 964GB / 966GB 9.07MB / 18.1MB 15
d6b272a420ca portal_tenant-provisioner.1.w8elgb8uw3557xy0aogwl9ibd 0.07% 814.9MiB / 2.197GiB 36.22% 20.3GB / 11.6GB 868kB / 0B 80
cf226cef07b7 portal_portal-enterprise.44f3a66ccxvnntntfchzjttfq.yrqfyvxpb2073ou5x6c33phy6 986.89% 2.705GiB / 8.807GiB 30.72% 1.12GB / 2.21GB 1.27MB / 129kB 198
6dfd39faef20 portal_pssg.44f3a66ccxvnntntfchzjttfq.va9579geto3nzflnrf33afek9 0.40% 2.649GiB / 8.807GiB 30.08% 652GB / 216GB 46.2MB / 180GB 163
7dd84cb6b774 portal_apim.44f3a66ccxvnntntfchzjttfq.h5f0qefpw8egavh9avvq7jsgu 0.53% 3.165GiB / 8.807GiB 35.94% 767GB / 560GB 30.7MB / 213GB 582
4cdf52a39693 portal_rabbitmq.44f3a66ccxvnntntfchzjttfq.5ug3pltaqkevly5ot2ng4r6u5 0.66% 207.4MiB / 1.099GiB 18.43% 7.92GB / 4.21GB 30.8MB / 296MB 237
a84099f51d9b portal_portaldb-slave.1.ehmg23yzzi1deplx3232wn288 0.06% 2.328MiB / 62.76GiB 0.00% 120GB / 19.2GB 65.7MB / 1.13TB 6
df9789fb77e6 portal_authenticator.44f3a66ccxvnntntfchzjttfq.7p3b02dwv6i2cz34qho04xpsi 0.06% 672.9MiB / 2.197GiB 29.91% 1.14GB / 1.32GB 0B / 0B 58
4d80ff93459a portal_historical.1.wctaldrmq3vibqs8q5lreexww 0.23% 2.993GiB / 5GiB 59.86% 10.9GB / 459MB 3.57TB / 33.2GB 149
7a82dfab41b0 portal_ingestion-server.1.hz7xkt8a76eyvtitlcls9e1we 0.07% 650.6MiB / 1.5GiB 42.35% 252GB / 1.62TB 0B / 0B 51
886545000ea1 portal_broker.1.m364a96og9k408v9b5dfe7nox 0.39% 1.552GiB / 1.562GiB 99.31% 1.26GB / 2.05GB 0B / 0B 180
4d3b7301c72d portal_kafka.1.ogbfw1jm2nb5ug6unvqqw3nx2 0.46% 1.132GiB / 1.5GiB 75.46% 3.28TB / 6.48TB 3.31GB / 181GB 86
8089fd797263 portal_coordinator.44f3a66ccxvnntntfchzjttfq.sx2ptcsvbux0rh4hgpju7qiae 31.16% 490.5MiB / 512MiB 95.79% 4.43TB / 33.1GB 3.52MB / 0B 163
2a458244617a portal_zookeeper.44f3a66ccxvnntntfchzjttfq.s2byuf2fwzscngm9arx7b66y5 0.06% 294.2MiB / 512MiB 57.46% 4.07GB / 31.4GB 14.1MB / 3.61GB 18
a595481962f9 portal_minio.1.hzxba6ywyesqmblj3ast9cwz2 0.00% 12.71MiB / 256MiB 4.97% 9.65GB / 9.61GB 554MB / 674MB 23
d183a0e542fa portal_portal-data.44f3a66ccxvnntntfchzjttfq.d0oicfitwqbmd8jfl26oga9te 0.06% 2.328GiB / 8.807GiB 26.44% 8.12GB / 9.16GB 1.16MB / 0B 183
810ebe92f0d6 appd_machine-agent.44f3a66ccxvnntntfchzjttfq.utip97oial8z5yi400upp7gxz 0.08% 740.2MiB / 62.76GiB 1.15% 10.8GB / 19.1GB 2.03MB / 0B 65
Release : 4.5
Unable to determine the root cause, need data while the issue occurring
Should this occur again collect the logs as the issue occurs
Good command to monitor
# docker stats --all --format "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
Collect logs
APIM home util/retrieve-logs.sh
Got inti the logs folder tar/gz the new folder created with all the collect logs/file:
example:
# /home/centos/apim-portal-5.0.2.3-final/util/logs/tar -czvf logs.tar.gz 2022-12-19-07-08-03
Resolved by restarting portal deployment