Connection multiplexing fails if remote server has idle timeout equal to interval between http requests.Using external monitoring system which is polled every 30 seconds and the connection would fail with a 503 error every now and then.
This occurs when the backend-server closes the connection due to an idle timeout .When the FIN/ACK is received from the server at the same time as the next request from the client is about to be forwarded AVI returns a 503 to the client.
By Default upstream_connpool_conn_idle_tmo
on Avi is set to 60 s or 1 Min however server timeout can be less that that.(For eg: 30sec)
Workaround is to reduce the upstream_connpool_conn_idle_tmo
to be less than 30 sec. Reducing the pool’s conn_idle_tmo
can have the connection closed by SE before server closes it due to server timeout. This way we can pick up a different connection from conn pool or just open a new one (if there is no open connect in conn pool for that server).
This is not a bug as frontend and backend connection are stiched to each other and we got a request at the same time when we are in the process of clearing the connection due to FIN from server side.