Two separate gpfdist processes bound to the same port number.
Article ID: 296867
Updated On:
Products
VMware Tanzu Greenplum
Issue/Introduction
After upgrading from 6.15 greenplum_client tools to 6.22 greenplum_client tools this issue was identified. The Customer starts multiple gpfdist processes on a Linux client. Two of the gpfdist processes bind to the same port which should not be the case.
This is the gpfdist log from one of the processes
This is the gpfdist log from one of the processes
2023-02-15 10:29:26 21242 INFO IPV6 socket: [::]:8011 2023-02-15 10:29:26 21242 INFO IPV4 socket: 0.0.0.0:8011 2023-02-15 10:29:26 21242 INFO Trying to open listening socket: 2023-02-15 10:29:26 21242 INFO IPV6 socket: [::]:8011 2023-02-15 10:29:26 21242 INFO Opening listening socket succeeded 2023-02-15 10:29:26 21242 INFO Trying to open listening socket: 2023-02-15 10:29:26 21242 INFO IPV4 socket: 0.0.0.0:8011 2023-02-15 10:29:26 21242 WARN listen with queue size 256 on socket (7) using port 8011 failed with error code (98): Address already in use Serving HTTP on port 8011, directory /home/dsadm/DS_Problemen/gpfdist/double_ports/tmp 2023-02-15 11:11:06 21242 WARN signal 15 received. gpfdist exits 2023-02-15 11:11:06 21242 INFO --------------------------------------- 2023-02-15 11:11:06 21242 INFO STATUS: total session(s) 0
This is the log from another gpfdist process
2023-02-15 10:29:26 21242 INFO IPV6 socket: [::]:8011 2023-02-15 10:29:26 21242 INFO IPV4 socket: 0.0.0.0:8011 2023-02-15 10:29:26 21242 INFO Trying to open listening socket: 2023-02-15 10:29:26 21242 INFO IPV6 socket: [::]:8011 2023-02-15 10:29:26 21242 INFO Opening listening socket succeeded 2023-02-15 10:29:26 21242 INFO Trying to open listening socket: 2023-02-15 10:29:26 21242 INFO IPV4 socket: 0.0.0.0:8011 2023-02-15 10:29:26 21242 WARN listen with queue size 256 on socket (7) using port 8011 failed with error code (98): Address already in use Serving HTTP on port 8011, directory /home/dsadm/DS_Problemen/gpfdist/double_ports/tmp 2023-02-15 11:11:06 21242 WARN signal 15 received. gpfdist exits 2023-02-15 11:11:06 21242 INFO --------------------------------------- 2023-02-15 11:11:06 21242 INFO STATUS: total session(s) 0
Environment
Product Version: 6.22
Resolution
A service server will call bind() method which binds it to a specific IP and port so that it can listen to incoming requests on that IP and port.
It will then call listen() method which puts the server into listen mode. This allows the server to listen to incoming connections
If a gpfdist is bound and listening on port 8000(ipv6) and is only bound on port 8000(ipv4),
At this moment the OS switches to another gpfdist. The second gpfdist skips port 8000(ipv6), and binds on port 8000(ipv4), and listens to port 8000(ipv4).
When the OS switches back to the first gpfdist, It will find that port 8000(ipv4) has been used. As gpfdist is listening on a port, that is port 8000(ipv6) it will continue and not error out. In this case, the two gpfdist listen on the same port.
The core of the problem is, socket interface 'bind' is not exclusive, that is 2 or more processes can bind to the same port. However, 'listen' is exclusive, that is only one process can listen on a port. gpfdist does not notice that there is another gpfdist trying to compete to listen on the same port.
Coding fix has been released in Greenplum 6.24.0 and above.
It will then call listen() method which puts the server into listen mode. This allows the server to listen to incoming connections
If a gpfdist is bound and listening on port 8000(ipv6) and is only bound on port 8000(ipv4),
At this moment the OS switches to another gpfdist. The second gpfdist skips port 8000(ipv6), and binds on port 8000(ipv4), and listens to port 8000(ipv4).
When the OS switches back to the first gpfdist, It will find that port 8000(ipv4) has been used. As gpfdist is listening on a port, that is port 8000(ipv6) it will continue and not error out. In this case, the two gpfdist listen on the same port.
The core of the problem is, socket interface 'bind' is not exclusive, that is 2 or more processes can bind to the same port. However, 'listen' is exclusive, that is only one process can listen on a port. gpfdist does not notice that there is another gpfdist trying to compete to listen on the same port.
Coding fix has been released in Greenplum 6.24.0 and above.
Feedback
Yes
No
Powered by
