portaldb deployment failed with misleading "invalid memory" error in CA API developer portal 4.2

Products

• CA API Developer Portal
• CA API Gateway

Issue/Introduction

Log of the portaldb or portaldb-slave container shows error "FATAL: invalid memory request size xxxx", but the docker host still has enough memory.
The portaldb/portaldb-slave are using default postgres database.

Environment

Release:
Component: APIPRD

Resolution

The error message could be misleading. In fact, you may see another HINT message in the log,
portal_portaldb-slave.1.xxx    | HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.

For portal 4.2, the memory limit of portaldb containers are set as the same as the RAM of docker host. ie. as soon as docker host still has free memory, the "invalid memory" message is more likely due to the corrupted data. In that case, we may need to restore the database with previous backup. 

For more details about backup/restore, please refer to,

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/4-5/set-up-and-maintenance/maintain-the-api-developer-portal/back-up-and-restore-internal-database.html

Here are some commands may be helpful on trouble shooing,
1. check memeory usage
free -h 
docker stats --no-stream 

2. export portaldb logs
1). docker service logs portal_portaldb-slave > portal_portaldb-slave.log 2>&1 
2). docker service logs portal_portaldb > portal_portaldb.log 2>&1 

 

Additional Information

Some discussions on postgres,
https://www.postgresql.org/message-id/54889560.6000501%40fuzzy.cz
https://www.postgresql.org/message-id/20160212074625.a8255b924c07c3f716dc191a%40potentialtech.com