Large file uploads through an NSX L7 LB fail

Products

• VMware NSX

Issue/Introduction

• Transferring files between client and backend servers through a L7 Load Balancer fail with a 500 error
• On the Edge log /var/log/syslog, an error like this example is observed:
  
  <DATE_TIME> <HOSTNMAE> NSX <PID> LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB UUID>] pwrite() "/opt/vmware/nsx-edge/lb/client_body_temp/0007347903" failed (28: No space left on device), client: <CLIENT_IP_ADDRESS>, server: , request: <REST OF REQUEST BODY>

Environment

VMware NSX-T Data Center

VMware NSX

Cause

This issue is caused as nginx for L7 Load Balancer buffers a client request body to a file under /opt/vmware/nsx-edge/lb/client_body_temp directory, as a result of which, the file that is being uploaded to the backend server is first temporarily uploaded to the edge load balancer under root partition.
If there's no available space for /opt/vmware/nsx-edge/lb/client_body_temp directory on edge, buffering (caching/temporary upload) of the file will fail with "No space left on device" error message of /var/log/syslog on an edge node, resulting in the failure of the upload.

 

Resolution

This is expected L7 Load Balancer behavior that the request body needs to be cached for transferring it to the backend server.
The below workarounds are available to upload the file to the backend server:

• Use L4 load balancer instead of the L7 load balancer.
  In the case of L4 LB, the file does not need to be cached on the edge.
• Reduce the size of the file that is being transferred