The message "Internal server error" is recorded in the LbAccessLog.

Products

• VMware NSX

Issue/Introduction

It is observed in /var/log/syslog that Error.Reason is "Internal server error".

<DATE_TIME> <HOSTNAME> NSX 3176155 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="access" level="INFO"] [<LB_UUID>][<VS_UUID>] Operation.Category: 'LbAccessLog', Operation.Type: 'TCP', Lb.UUID: '<LB_UUID>', Lb.Name: '<LB_NAME>', Vs.UUID: '<VS_UUID>', Vs.Name: '<VS_NAME>', Vs.Ip: '<IP>', Vs.Port: '<PORT>', Pool.UUID: '<POOL_UUID>', Pool.Name: '<POOL_NAME>', PoolMember.IP: '<IP>', PoolMember.Port: '<PORT>', Client.Ip: '<IP>', Client.Port: '<PORT>', Snat.Ip: '<IP>', Snat.Port: '<PORT>', Error.Reason: 'Internal server error'

 

"stale create req error" and "pool empty error" would be reported in /edge/lb_<LB_UUID>/l4cp from the support bundle.

"stale create req error": 1632033, "server selection failed error": 5763026, "snat failed error": 502, "out of conn error": 3163716, "bad CP conn ID error": 1683025, "stale delete req error": 13113 "pool empty error": 4838

Environment

VMware NSX

Cause

It could be caused due to lack of available message queue in the load balancer compared to heavy traffic in case of using the load balancer with SMALL size.

The size of available message queue depends on the size of the load balancer.

Resolution

This issue could be addressed as increasing the size of the load balancer to MEDIUM size.

We don't recommend to use the load balancer with SMALL size in production environment.

https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/load-balancer/key-load-balancer-concepts/scaling-load-balancer-resources.html