unable to start production server after shared_buffers configuration change
Article ID: 296391
Updated On:
Products
Issue/Introduction
- after configuration change in postgresql.conf form
shared_buffers = 16GB
to
shared_buffers = 32GBdatabase unable to start
$ /usr/bin/pg_ctl -D /pgdata/12/ start waiting for server to start....time=2022-11-05 03:38:28 EDT, pid=55696 LOG: starting PostgreSQL 12.5 (VMware Postgres 12.5.0) on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit time=2022-11-05 03:38:28 EDT, pid=55696 LOG: listening on IPv4 address "0.0.0.0", port 5432 time=2022-11-05 03:38:28 EDT, pid=55696 LOG: listening on IPv6 address "::", port 5432 time=2022-11-05 03:38:28 EDT, pid=55696 LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" time=2022-11-05 03:38:28 EDT, pid=55696 FATAL: could not map anonymous shared memory: Cannot allocate memory time=2022-11-05 03:38:28 EDT, pid=55696 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 35577274368 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections. time=2022-11-05 03:38:28 EDT, pid=55696 LOG: database system is shut down stopped waiting pg_ctl: could not start server
Environment
Resolution
#huge_pages = try
- found huge pages in postgresql.auto.conf
cat /pgdata/12/postgresql.auto.conf | grep huge_pages huge_pages = 'on'
This means system doesn’t have enough space reserved for huge pages. As a rule of thumb, if you explicitly want to have shared_buffers in huge pages you need to reserve those using vm.nr_hugepages. You can’t simply raise shared buffers.
For more details on how to properly setup the server for that have a look on: https://www.postgresql.org/docs/12/kernel-resources.html#LINUX-HUGE-PAGES
You need to manually compute the number of huge pages. Let's check the current configuration:
[postgres@master ~]$ grep Huge /proc/meminfo AnonHugePages: 0 kB HugePages_Total: 12288 HugePages_Free: 3786 HugePages_Rsvd: 75 HugePages_Surp: 0 Hugepagesize: 2048 kB [postgres@master ~]$
From above we can see OS has 12288 pages with size 2048 kB so together the OS reserved 12288 * 2048 = 25165824kb = 25.16GB
If you want to increase shared buffers to 32 GB we missing 8 more gb in reserved memory. Let's have alook how much memory we have left
[postgres@master ~]$ free -m ip total used free shared buff/cache available Mem: 128755 28772 1391 9 98591 99277 Swap: 4095 26 4069
As you see form above we only have 1391mb free memory and we need 8gb. In this situation this cluster is not able to reserve this memory because it doesn't have enough physical memory. Fortunately this host is a VM and all Customer had to do is to increase the memory in VM and this was reflected in host OS after restart
After you confirm you have enough of free memory you can increase the reserved shared buffer by forcing OS to reserve this memory with editing /proc/sys/vm/nr_hugepages
(since we don't have example from live Customer environment please find a lab example below)
lab example, increasing shared_buffers by 10mb:
[root@mdw-lab2 ~]# free -m
total used free shared buff/cache available
Mem: 7821 1337 1017 1570 5466 4834
Swap: 2047 60 1987
[root@mdw-lab2 ~]# cat /proc/sys/vm/nr_hugepages
70
[root@mdw-lab2 ~]# grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 70
HugePages_Free: 70
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@mdw-lab2 ~]# echo 75 > /proc/sys/vm/nr_hugepages
[root@mdw-lab2 ~]# cat /proc/sys/vm/nr_hugepages
75
[root@mdw-lab2 ~]# free -m
total used free shared buff/cache available
Mem: 7821 1347 1007 1570 5466 4824
Swap: 2047 60 1987
From above example.
1) we have 1017mb free memory
2) there is 70 huge pages reserved so 70 * 2048kb = 143.36Mb
3) we force OS to reserve 5 more pages with echo 75
4) 5 pages * 2048kb = 10mb so we used 10mb of free memory to do it
5) now when we check the free -m we can see 1017-1007=10 so 10Mb less. This 10Mb was used for our reservation.
Please note:
- if you force to reserve more then OS can reserve you will only reserve until the free memory runs out
- reserving memory might take few seconds to a minute especially if you reserving GB.
Feedback
