search cancel

CPU Utilization is critical on Portal Prod Server

book

Article ID: 256476

calendar_today

Updated On:

Products

CA API Developer Portal

Issue/Introduction

CPU Utilization is critical on Portal Prod Server. Server is currently using more than 80% CPU. docker stats is showing the portal_enterprise is consuming most of the CPU. Need your help here: - 

CONTAINER ID        NAME                                                                           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
5c2b1f5c4dff        portal_portaldb.1.sr3mz11rluwavalm6z4mofoao                                    6.04%               182MiB / 62.76GiB     0.28%               8.21GB / 273GB      4.06MB / 101GB      92
669519a12a8d        portal_middlemanager.1.w9bsb03rp5ffu466o7jk9tyic                               1.06%               2.643GiB / 4GiB       66.07%              4.29TB / 35.7GB     467MB / 1.69GB      385
9df28f80579c        portal_ingestion-server.2.7aspfo213d8xvnidlkpsige2k                            0.07%               655.9MiB / 1.5GiB     42.70%              251GB / 1.61TB      0B / 0B             51
c8a8d158c526        portal_analytics-server.44f3a66ccxvnntntfchzjttfq.2i65uvppvpyc9tkgm0ptv8r00    0.06%               431MiB / 4.4GiB       9.56%               313MB / 216MB       1.88MB / 0B         83
6bc663bb1d57        portal_dispatcher.44f3a66ccxvnntntfchzjttfq.ikzu77ipu7cngdp7tlqyly22l          0.06%               22.99MiB / 1.099GiB   2.04%               964GB / 966GB       9.07MB / 18.1MB     15
d6b272a420ca        portal_tenant-provisioner.1.w8elgb8uw3557xy0aogwl9ibd                          0.07%               814.9MiB / 2.197GiB   36.22%              20.3GB / 11.6GB     868kB / 0B          80
cf226cef07b7        portal_portal-enterprise.44f3a66ccxvnntntfchzjttfq.yrqfyvxpb2073ou5x6c33phy6   986.89%             2.705GiB / 8.807GiB   30.72%              1.12GB / 2.21GB     1.27MB / 129kB      198
6dfd39faef20        portal_pssg.44f3a66ccxvnntntfchzjttfq.va9579geto3nzflnrf33afek9                0.40%               2.649GiB / 8.807GiB   30.08%              652GB / 216GB       46.2MB / 180GB      163
7dd84cb6b774        portal_apim.44f3a66ccxvnntntfchzjttfq.h5f0qefpw8egavh9avvq7jsgu                0.53%               3.165GiB / 8.807GiB   35.94%              767GB / 560GB       30.7MB / 213GB      582
4cdf52a39693        portal_rabbitmq.44f3a66ccxvnntntfchzjttfq.5ug3pltaqkevly5ot2ng4r6u5            0.66%               207.4MiB / 1.099GiB   18.43%              7.92GB / 4.21GB     30.8MB / 296MB      237
a84099f51d9b        portal_portaldb-slave.1.ehmg23yzzi1deplx3232wn288                              0.06%               2.328MiB / 62.76GiB   0.00%               120GB / 19.2GB      65.7MB / 1.13TB     6
df9789fb77e6        portal_authenticator.44f3a66ccxvnntntfchzjttfq.7p3b02dwv6i2cz34qho04xpsi       0.06%               672.9MiB / 2.197GiB   29.91%              1.14GB / 1.32GB     0B / 0B             58
4d80ff93459a        portal_historical.1.wctaldrmq3vibqs8q5lreexww                                  0.23%               2.993GiB / 5GiB       59.86%              10.9GB / 459MB      3.57TB / 33.2GB     149
7a82dfab41b0        portal_ingestion-server.1.hz7xkt8a76eyvtitlcls9e1we                            0.07%               650.6MiB / 1.5GiB     42.35%              252GB / 1.62TB      0B / 0B             51
886545000ea1        portal_broker.1.m364a96og9k408v9b5dfe7nox                                      0.39%               1.552GiB / 1.562GiB   99.31%              1.26GB / 2.05GB     0B / 0B             180
4d3b7301c72d        portal_kafka.1.ogbfw1jm2nb5ug6unvqqw3nx2                                       0.46%               1.132GiB / 1.5GiB     75.46%              3.28TB / 6.48TB     3.31GB / 181GB      86
8089fd797263        portal_coordinator.44f3a66ccxvnntntfchzjttfq.sx2ptcsvbux0rh4hgpju7qiae         31.16%              490.5MiB / 512MiB     95.79%              4.43TB / 33.1GB     3.52MB / 0B         163
2a458244617a        portal_zookeeper.44f3a66ccxvnntntfchzjttfq.s2byuf2fwzscngm9arx7b66y5           0.06%               294.2MiB / 512MiB     57.46%              4.07GB / 31.4GB     14.1MB / 3.61GB     18
a595481962f9        portal_minio.1.hzxba6ywyesqmblj3ast9cwz2                                       0.00%               12.71MiB / 256MiB     4.97%               9.65GB / 9.61GB     554MB / 674MB       23
d183a0e542fa        portal_portal-data.44f3a66ccxvnntntfchzjttfq.d0oicfitwqbmd8jfl26oga9te         0.06%               2.328GiB / 8.807GiB   26.44%              8.12GB / 9.16GB     1.16MB / 0B         183
810ebe92f0d6        appd_machine-agent.44f3a66ccxvnntntfchzjttfq.utip97oial8z5yi400upp7gxz         0.08%               740.2MiB / 62.76GiB   1.15%               10.8GB / 19.1GB     2.03MB / 0B         65

Environment

Release : 4.5

Cause

Unable to determine the root cause, need data while the issue occurring 

Should this occur again collect the logs as the issue occurs 

Good command to monitor

# docker stats --all --format "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

Collect logs 

APIM home  util/retrieve-logs.sh

 

Got inti the logs folder tar/gz the new folder created with all the collect logs/file:

example:

# /home/centos/apim-portal-5.0.2.3-final/util/logs/tar -czvf logs.tar.gz 2022-12-19-07-08-03

 

Resolution

Resolved by restarting portal deployment

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/4-5/install-configure-and-upgrade/install-portal-on-docker-swarm/troubleshoot-the-installation.html#concept.dita_acf7d59b989ca9e023cf6543e787ad6907001961_RestartDeployment