Pool Members DOWN after upgrading edge to 6.4.4 or higher. HTTP/HTTPS service monitor not working with host header extension.

Products

• VMware NSX

Issue/Introduction

Symptoms:

• After upgrading edge to 6.4.4 or higher, load balancer pool members are DOWN.
• An HTTP or HTTPS service monitor is used.
• Attempting to access the backend servers directly via browser/curl proves that the servers are working as expected however they fail when accessed via LB
• Requests may fail with SSL_ERROR_SYSCALL
• In the extension field of the service monitor, a host header is specified using this format:

header="Host:host.domain.xyz"

• That host header is not included in the HTTP/HTTPS request as it should.

Example from an HTTP request where the host header is correctly included pre-6.4.4.

Frame 3: 213 bytes on wire (1704 bits), 213 bytes captured (1704 bits)
Ethernet II, Src: Vmware_88:6e:cb (##:##:##:##:##:cb), Dst: Vmware_88:2d:d9 (##:##:##:##:##:d9)
Internet Protocol Version 4, Src: 172.16.1.1, Dst: 172.16.1.100
Transmission Control Protocol, Src Port: 12348, Dst Port: 80, Seq: 1, Ack: 1, Len: 159
Hypertext Transfer Protocol
    GET / HTTP/1.0\r\n
    User-Agent: check_http/v2.0.3 (nagios-plugins 2.0.3)\r\n
    Connection: close\r\n
    Accept: */*\r\n
    Host:host.domain.xyz\r\n
    X-Forwarded-Proto:https\r\n
    \r\n
    [Full request URI: http://host.domain.xyz/]
    [HTTP request 1/1]

Example from an HTTP request where the host header is skipped after upgrade to 6.4.4+.

Frame 3: 183 bytes on wire (1464 bits), 183 bytes captured (1464 bits)
Ethernet II, Src: Vmware_88:b6:33 (##:##:##:##:##:33), Dst: Vmware_88:99:83 (##:##:##:##:##:83)
Internet Protocol Version 4, Src: 172.16.1.1, Dst: 172.16.1.100
Transmission Control Protocol, Src Port: 21920, Dst Port: 80, Seq: 1, Ack: 1, Len: 129
Hypertext Transfer Protocol
    GET / HTTP/1.0\r\n
    User-Agent: check_http/v2.2.1 (nagios-plugins 2.2.1)\r\n
    Connection: close\r\n
    Accept: */*\r\n
    X-Forwarded-Proto:https\r\n
    \r\n
    [HTTP request 1/1]

Cause

Starting 6.4.4, nagios plugins were upgraded to 2.2.1, specifically check_http which is used by some http and https service monitors.

On this new version, check_http expects "-H <hostname>" after header=Host:<hostname> in the extension field to add Host header correctly in the request. Otherwise the host will not be included in the header and the request can fail.

Resolution

Edit the Service Monitor and update the extension to the new syntax.

Original:

header=Host:host.domain.xyz

Updated:

header=Host:host.domain.xyz -H host.domain.xyz

You should now see that the host header is correctly included in the request.

Frame 3: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits)
Ethernet II, Src: Vmware_88:b6:33 (##:##:##:##:##:33), Dst: Vmware_88:99:83 (##:##:##:##:##:83)
Internet Protocol Version 4, Src: 172.16.1.1, Dst: 172.16.1.100
Transmission Control Protocol, Src Port: 21982, Dst Port: 80, Seq: 1, Ack: 1, Len: 160
Hypertext Transfer Protocol
    GET / HTTP/1.1\r\n
    User-Agent: check_http/v2.2.1 (nagios-plugins 2.2.1)\r\n
    Connection: close\r\n
    Host: host.domain.xyz\r\n
    Accept: */*\r\n
    X-Forwarded-Proto:https\r\n
    \r\n
    [Full request URI: http://host.domain.xyz/]
    [HTTP request 1/1]