Greenplum Error :"FATAL", "53300", "Sorry, Too Many Clients Already"

Products

• VMware Tanzu Greenplum

Issue/Introduction

The Master log reports a FATAL message as below:

2016-10-19 03:27:03.712908 CST,"###","###",p####,th-989944032,"###.###.###.###","34541",2016-10-19 03:27:03 CST,0,,,seg-1,,,,,"FATAL","53300","sorry, too many clients already",,,,,,,0,,"proc.c",334,
2016-10-19 03:27:04.588362 CST,"xxx","xxx",p#####,th-989944032,"###.###.###.###","54642",2016-10-19 03:27:04 CST,0,,,seg-1,,,,,"FATAL","53300","sorry, too many clients already",,,,,,,0,,"proc.c",334,

Cause

There are two scenarios where this issue may occur:

1. The database is operating normally, but there more connections need to be established than max_connection parameter.
   
   The "max_connections" parameter may need to be adjusted to meet your requirements.
2. The database is experiencing an issue which is causing connections to queue up.
3. There are many different issues which may lead to max_connections being exceeded. We can start with below steps:

• • Check if any host has lots of startup process.
  • Check if master log reported any instance can't be connected.
  • Check if any instance had their postgres process reset or missing.
  • Check if all hosts can be connected and space usage(100% used?).
  • If RCA required, take core dump for one startup process before restart the db.

Resolution

1. If root cause required, then contact pivotal support before restart the db.
2. if root cause not needed, restart the db directly.