503 errors observed while communicating to mirror Server via sensor gateway
Article ID: 381271
Updated On:
Products
Issue/Introduction
When configuring the CBC sensors to download Signature updates from a mirror server and these sensors are managed by Sensor Gateway (SGW) using the hostname only of the mirror server instead of the FQDN, 503 errors (Unhealthy upstream) will be seen in SGW envoy-access.log files.
Example of these errors
admin@photon-machine [ ~ ]$ grep idx /opt/vmware/sgw/data/logs/envoy-* | grep 503
/opt/vmware/sgw/data/logs/envoy-access.log:[2024-10-15T13:11:18.342Z] 0 '<Source IP address>:<Source Port>' '<Destination Mirror server>' 'GET /<Base64_Hashed_URL>/idx/master.idx HTTP/1.1' 503 UH -
/opt/vmware/sgw/data/logs/envoy-access.log:[2024-10-15T13:24:41.120Z] 0 '<Source IP address>:<Source Port>' '<Destination Mirror server>' 'GET /<Base64_Hashed_URL>/idx/master.idx HTTP/1.1' 503 UH -
/opt/vmware/sgw/data/logs/envoy-access.log:[2024-10-15T13:25:52.107Z] 0 '<Source IP address>:<Source Port>' '<Destination Mirror server>' 'GET /<Base64_Hashed_URL>/idx/master.idx HTTP/1.1' 503 UH -
Example of errors received at the client side or SGW CLU
admin@photon-machine [ ~ ]$ curl -vv --noproxy '*' https://<SGW Entry Point URL>/<Base64_Hashed_URL>/idx/master.idx -k
* Trying <SGW Entry Point URL>:443...
* Connected to <SGW Entry Point URL> (<SGW Entry Point URL>) port 443
* ALPN: curl offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
* subject: CN=<SGW Entry Point URL>; OU=SBU; O=VMware; L=PaloAlto; ST=CA; C=US
* start date: Oct 15 08:08:29 2024 GMT
* expire date: Oct 15 08:08:29 2025 GMT
* issuer: CN=<SGW Entry Point URL>; OU=SBU; O=VMware; L=PaloAlto; ST=CA; C=US
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /<Base64_Hashed_URL>/idx/master.idx HTTP/1.1
> Host: <SGW Entry Point URL>
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Mon, 21 Oct 2024 10:49:46 GMT
< server: envoy
<
* Connection #0 to host <SGW Entry Point URL> left intact
no healthy
Environment
Sensor Gateway 1.2+
Cause
SGW has a docker container inside it, that contains the envoy component.
The search domain configured in the photon OS of SGW doesn't get copied to the docker container, hence envoy component sends the DNS requests trying to resolve the hostname of the mirror server, which results in failing the DNS queries which consequently results in not establishing the TCP connections with mirror server and returning 503 errors to CBC sensor.
Resolution
Use FQDN of mirror server instead of hostname.
http://localmirrorupdateserver.example.com is supported.
http://localmirrorupdateserver.com is supported.
http://localmirrorupdateserver is not supported.
Envoy component will be requesting the resolution of the FQDN of the mirror server.
Additional Information
Feedback
