Large file transfer(s) fail, when going through an NSX Load Balancer which has a layer 7 Virtual Server

Products

• VMware NSX

Issue/Introduction

• Transferring files between client and backend servers through a Layer 7 (L7) Virtual Server fail with a 500 error.
• Same transfer will succeed when using a Layer 4 (L4) Virtual Server.
• The file being transferred is larger than 12GB.
• The following log entries are seen on the active edge, where the Load Balancer with the Layer 7 Virtual Server resides, in the log /var/log/syslog:
  
  <DATE_TIME> <HOSTNMAE> NSX <PID> LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB UUID>] pwrite() "/opt/vmware/nsx-edge/lb/client_body_temp/0007347903" failed (28: No space left on device), client: <CLIENT_IP_ADDRESS>, server: , request: <REST OF REQUEST BODY>

Environment

VMware NSX-T Data Center

VMware NSX

Cause

This issue is caused as nginx for L7 Load Balancer caches a client request body to a file under /opt/vmware/nsx-edge/lb/client_body_temp directory, as a result of which, the file that is being uploaded to the backend server is first temporarily cached on the under root (/) partition.

If there is no available space on / directory on edge, caching of the file will fail with "No space left on device" error message of /var/log/syslog on an edge node, resulting in the failure of the upload.

Note: On a freshly deployed edge node, there is 12GB available space on the edge node / partition, if other files take up space on the / partition, this will reduce the size of the file which can be transferred.

To check available space on the / partition, as admin user, run: get filesystem-stats

Resolution

This is expected behavior, when using a L7 virtual server.

If you encounter this issue and you need to transfer large files, you need to use a L4 virtual server or reduce the file size.

Additional Information

If this KB did not help resolve your issue, you can review the following KB for further troubleshooting steps: Troubleshooting NSX Native Load Balancer