To resolve this issue, enable either active monitor or passive monitor on the default pool so that the L4 virtual server can re-dispatch the traffic to the sorry pool if no servers on default pool are available.
Note: Even for L7 virtual server, it is also recommended to enable monitor on the default pool so that load balancer will not retry every default backend servers for each HTTP request so that the traffic could be re-dispatched to sorry pool.