Large file transfer(s) fail, when going through an NSX Load Balancer which has a layer 7 Virtual Server

Products

• VMware NSX

Issue/Introduction

• Transferring files between client and backend servers through a Layer 7 (L7) Virtual Server fail with a http 500 error.
• Same transfer will succeed when using a Layer 4 (L4) Virtual Server.
• The file being transferred is larger than 12GB or multiple files are being simultaneously transferred with a combined size greater than 12GB.
• The following log entries are seen on the active edge, where the Load Balancer with the Layer 7 Virtual Server resides, in the log /var/log/syslog:
  
  <DATE_TIME> <HOSTNMAE> NSX <PID> LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB UUID>] pwrite() "/opt/vmware/nsx-edge/lb/client_body_temp/0007347903" failed (28: No space left on device), client: <CLIENT_IP_ADDRESS>, server: , request: <REST OF REQUEST BODY>

<HOSTNAME> NSX <PID> LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="access" level="INFO"] [<UUID>][<UUID>] Operation.Category: 'LbAccessLog', Operation.Type: 'Http', Lb.UUID: '<LB UUID>', Lb.Name: '<LB NAME>', Vs.UUID: '<VS UUID>', Vs.Name: '<VS NAME>', Vs.Ip: '<VS IP>', Vs.Port: 'TCP PORT', Pool.UUID: '-', Pool.Name: '-', PoolMember.Ip: '-', PoolMember.Port: '-', Client.Ip: '<CLIENT IP>', Client.Port: '<CLIENT PORT>', Snat.Ip: '-', Snat.Port: '-', HttpRequest.Method: 'PUT', HttpRequest.UserAgent: 'curl/7.47.1', HttpRequest.X-Fwd-For: '-', HttpRequest.Uri: '/<PATH TO DIRECTORY/FILENAME.FILETYPE', HttpRequest.Host: '<HOSTNAME>', HttpResponse.Status: '500', HttpResponse.StatusCategory: '5xx', HttpResponse.Size: '-', HttpResponse.ServerTime: '-', HttpResponse.TotalTime: '52.988', Error.Reason: '-'

Environment

VMware NSX-T Data Center

VMware NSX

Cause

This issue is caused as nginx for L7 Load Balancer caches a client request body to a file under /opt/vmware/nsx-edge/lb/client_body_temp directory, as a result of which, the file that is being uploaded to the backend server is first temporarily cached under the root (/) partition.

If there is no available space on / directory on edge, caching of the file will fail with "No space left on device" error message of /var/log/syslog on an edge node, resulting in the failure of the upload.

Note: On a freshly deployed edge node, there is 12GB available space on the edge node / partition, if other files take up space on the / partition, this will reduce the size of the file which can be transferred.

To check available space on the / partition, as admin user, run: get filesystem-stats

Resolution

This is expected behavior, when using a L7 virtual server.

If you encounter this issue and you need to transfer large files, you need to use a L4 virtual server or reduce the file size.

Additional Information

If this KB did not help resolve your issue, you can review the following KB for further troubleshooting steps: Troubleshooting NSX Native Load Balancer