Errors in logs from proxy servers noting invalid requests between vCenter and proxy server

Products

• VMware vCenter Server

Issue/Introduction

In environments where the vCenter uses a proxy log events like those bellow are noted in the proxy logs

Example events from proxy logging 

Date xxx  xx.xx.xxx.xxx NONE_NONE/xxx 3812 - error:invalid-request - HIER_NONE/- text/html
Date xxx xx.xx.xxx.xxx NONE_NONE/xxx 3811 - error:invalid-request - HIER_NONE/- text/html
Date xxx xx.xx.xxx.xxx NONE_NONE/xxx 3810 - error:invalid-request - HIER_NONE/- text/html
Date xxx xx.xx.xxx.xxx NONE_NONE/xxx 3811 - error:invalid-request - HIER_NONE/- text/html 
Date xxx0 xx.xx.xxx.xxx NONE_NONE/xxx 3811 - error:invalid-request - HIER_NONE/- text/html

Environment

vCenter 7.x

vCenter 8.x

Cause

This is an issue at the proxy server layer

When using a tcpdump to create a pcap we can match the errors in the proxy logs to expected vCenter telemetry data queries made over the network

Example pcap command that writes output to a file

tcpdump -i any -npvvvX -s 65535 'host <proxy IP>' vcenter.pcap

In this pcpa we will see events like the ones bellow between the vCenter IP and the proxy IP

• 273    xxx.xxxxxxx    xx.xx.xxx.xx    xxx.xxx.xxx.xx    HTTP    577    1    Continuation.
• 285    xxx.xxxxxxx xx.xx.xxx.xx    xxx.xxx.xxx.xx    HTTP    100    1    CONNECT vcsa.vmware.com:443 HTTP/1.0 .
• 287    xxx.xxxxxxx xx.xx.xxx.xx    xxx.xxx.xxx.xx    HTTP    99    41    HTTP/1.1 200 Connection established 
• 303    xxx.xxxxxxx xx.xx.xxx.xx    xxx.xxx.xxx.xx    HTTP    577    1    Continuation

 

These telemetry events will align with the errors noted in the proxy logging 

These events are from the analytics service that will try to keep the connection to Broadcom telemetry server by sending KEEP ALIVE messages and handshakes in periodic time interval and are working as intended 

 

 

 

Resolution

As these telemetry events are expected this issue needs to be reviewed from the proxy side to determine why the events from the vCenter are logged and determined to be "error:invalid-request"