Portal crashing daily after upgrade to 5.1 on a default centos image

Products

• CA API Developer Portal

Issue/Introduction

After upgrading to portal 5.1 the portal is crashing daily 

In this case all pods were restarted. Here are the results for Developer Portal that has been running for 3 hours.

Docker stats

CONTAINER ID        NAME                                                                           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS

4b18fde2e023        portal_dispatcher.o4allbbjivhknfvglnt9dwj7g.5pn073plz152dln7623ledh2p          0.00%               8.961MiB / 560MiB     1.60%               96.1MB / 91.6MB     8.64MB / 0B         11

56c106995559        portal_portal-enterprise.o4allbbjivhknfvglnt9dwj7g.mk37rv1uilej8lhyly362hyz5   0.16%               1.243GiB / 3.839GiB   32.39%              11.3MB / 3.21MB     460MB / 64.5kB      64

96676cdc438f        portal_tenant-provisioner.1.nshfbljj22lhs1j3f835xxjaa                          0.47%               497.7MiB / 1.095GiB   44.40%              37.1MB / 16.1MB     172MB / 0B          58

8cb05ee6b13c        portal_analytics-server.o4allbbjivhknfvglnt9dwj7g.6g2p5lswbzotautozymv5fg3h    0.15%               381.9MiB / 2.191GiB   17.02%              1.43MB / 362kB      217MB / 0B          64

d7a53a188159        portal_historical.2.ne6exys0iybm5gonigfv8ekww                                  0.26%               2.318GiB / 5GiB       46.36%              777MB / 8.06MB      9.84GB / 4.49GB     135

94feaa44f264        portal_pssg.o4allbbjivhknfvglnt9dwj7g.kfjw7vcdh153f9trs450iz496                0.66%               4.941GiB / 6.579GiB   75.11%              74.8MB / 51.7MB     561MB / 271MB       181

eeedf3d3f33f        portal_smtp.1.qjkqfgop7vyi16731tiwatys7                                        0.02%               11.91MiB / 560MiB     2.13%               8.22kB / 0B         18.5MB / 556kB      12

5e6f18fa606e        portal_zookeeper.o4allbbjivhknfvglnt9dwj7g.gllfeodl2zb6796y6iw11d8a9           0.63%               142.4MiB / 512MiB     27.81%              10.4MB / 48.1MB     59.3MB / 75.7MB     48

13ed87e98ba5        portal_ingestion-server.1.087o42pjtia2x5hjo9ewy52mt                            0.15%               233.6MiB / 1.5GiB     15.21%              61.4MB / 479MB      104MB / 0B          34

c79958344833        portal_apim.o4allbbjivhknfvglnt9dwj7g.j48qlyel9yxrbbuc5if9bors3                0.62%               5.258GiB / 6.579GiB   79.91%              51.7MB / 49.7MB     442MB / 987MB       217

822f12e0e70b        portal_portal-data.o4allbbjivhknfvglnt9dwj7g.dnv6k7re0wcqaxplslpiy7ir5         4.44%               1.184GiB / 3.839GiB   30.84%              55.5MB / 43.8MB     242MB / 0B          88

a68dab10afbd        portal_middlemanager.1.ebzagmqkyd3u4qk35gzqvh1tq                               1.19%               3.196GiB / 4GiB       79.89%              3.28GB / 34.8MB     129MB / 2.85MB      366

0bfb0a80f1fd        portal_authenticator.o4allbbjivhknfvglnt9dwj7g.dgt3xsc54aowt8h74wdwc2mdh       0.06%               297MiB / 1.095GiB     26.50%              3.6MB / 2.27MB      159MB / 0B          47

e93dbc1cd2e6        portal_broker.1.wehxk25w8surxyxzbgen3m5r6                                      4.77%               1.463GiB / 1.562GiB   93.60%              7.66MB / 4.62MB     75.4MB / 0B         173

729d9711a87f        portal_kafka.1.j6f8n0ilp9zi1j478vhab4zjl                                       0.83%               575.5MiB / 1.5GiB     37.47%              490MB / 3.28GB      2.03GB / 101MB      86

195620fd0556        portal_rabbitmq.o4allbbjivhknfvglnt9dwj7g.d32sc7l9946f5jmpc79soysly            2.55%               115.2MiB / 560MiB     20.56%              885kB / 780kB       66MB / 2.6MB        164

ac6971cf0d2c        portal_coordinator.o4allbbjivhknfvglnt9dwj7g.rfyo5s8zjbhp2wr4x2ld4roci         4.70%               460.9MiB / 512MiB     90.03%              2.28GB / 13.6MB     112MB / 0B          156

f9d2efbb3ffe        portal_minio.1.v7ma0pjqsfs3fr1wwvewkl07x                                       1.61%               15.56MiB / 256MiB     6.08%               61.8MB / 1.57GB     104MB / 47.8MB      21

8440225a8067        portal_historical.1.3w00y5up28tlaiz412nxqwerp                                  0.26%               2.173GiB / 5GiB       43.46%              777MB / 8.05MB      9.67GB / 4.49GB     135

 

Environment

Release : 5.1

Resolution

The default centos image has 32gb , when scalling certaing containers to run multiple replica's 

for example the portal_historical container the default memory size of 32gb could be to small causing frequent crashes and restarts of containers.

if that turns out to be the case, the additional  historical container add another 2.5 gb to the used memory .  

To run multiple historical container you need increase the vm memory with around 4 gb .