Cloud Director OVF upload fails with error Broken pipe (Write failed)
Article ID: 320525
Updated On:
Products
Issue/Introduction
Symptoms:
- OVF upload fails with below error in Cloud Director
Error - Cannot retry request with a non-repeatable request entity - Broken pipe (Write failed)
- In the vCenter UI, we will see events similar to:
The object 'vim.VirtualMachine:vm-123' has already been deleted or has not been completely created
- In the vcloud-director-info.log or vcloud-director-debug.log on Cloud Director, you will find entries similar to:
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:108)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
... 13 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(SSLSocketOutputRecord.java:340)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1299)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
- In the /var/log/vmware/vpxd/vpxd.log on vCenter, you will find entries similar to:
2022-09-05T03:16:36.865Z error vpxd[07893] [Originator@6876 sub=VAppImport opID=vcd-<OP_UUID>;activity=urn:uuid:<ACTIVITY_UUID>] Caught exception while im
porting VM: N5Vmomi5Fault15RequestCanceled9ExceptionE(Fault cause: vmodl.fault.RequestCanceled
--> )
- In the /var/run/log/vpxa.log on the ESXi host, you will find entries similar to:
2022-09-05T03:16:31.448Z warning vpxa[2100588] [Originator@6876 sub=Default] Timed out writing HTTP response. : Read timeout after approximately 50000ms. Closing stream <io_obj p:0x0000008e7213c738, h:26, <TC
P '127.0.0.1 : 12000'>, <TCP '127.0.0.1 : 57230'>>
2022-09-05T03:16:31.448Z warning vpxa[2101545] [Originator@6876 sub=IO.Connection opID=SWI-75d48b59] Failed to read buffer from stream; <io_obj p:0x0000008e7213c738, h:-1, <TCP '127.0.0.1 : 12000'>, <TCP '127
.0.0.1 : 57230'>> e: 2(End of file), async: false, duration: 47891msec
2022-09-05T03:16:31.448Z error vpxa[2101545] [Originator@6876 sub=Default opID=SWI-75d48b59] Error on read, error: -1
2022-09-05T03:16:31.448Z error vpxa[2101545] [Originator@6876 sub=HttpNfcServer opID=SWI-75d48b59] [DiskUploadWorker] Error reading input: N7Vmacore11IOExceptionE(Error on read, error: -1)
2022-09-05T03:16:31.450Z warning vpxa[2101545] [Originator@6876 sub=HttpNfcServer opID=SWI-75d48b59] [DiskUploadWorker] Caught exception while sending error response Error on read, error: -1: N7Vmacore16Timeo
utExceptionE(Operation timed out: Stream: <io_obj p:0x0000008e7213c738, h:-1, <TCP '127.0.0.1 : 12000'>, <TCP '127.0.0.1 : 57230'>>, duration: 00:00:47.934156 (hh:mm:ss.us))
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware Cloud Director 10.x
Cause
- During OVF upload, Cloud Director will send data to an ESXi host to upload it on the datastore.
- This issue occurs when the http connection times out on the ESXi host(Default timeout value is 50000ms).
Resolution
- To resolve this issue, increase the timeout value of vpxa agent on the ESXi hosts to 5 mins (600000ms) and retry the OVF upload.
For ESXi 6.x and prior to ESXi 7.0 U3, refer the below steps:
- Open a shell session to the ESXi host. For more information, see Using ESXi Shell in ESXi 6.x, 7.x and 8.x (2004746) .
- Take a backup of the file: /etc/vmware/vpxa/vpxa.cfg
For example:
cp /etc/vmware/vpxa/vpxa.cfg /etc/vmware/vpxa/vpxa.cfg.bak
- Edit the original vpxa.cfg file using a text editor: For more information, see Editing files on an ESXi host using vi (1020302) :
- Add <readTimeoutMs>600000</readTimeoutMs> and <handshakeTimeoutMs>600000</handshakeTimeoutMs> under http and ssl sections of vmacore respectively.
- Modified vpxa.cfg file should look like below:
<vmacore>
...
<http>
<defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
<readTimeoutMs>600000</readTimeoutMs>
</http>
...
<ssl>
<doVersionCheck>false</doVersionCheck>
<handshakeTimeoutMs>600000</handshakeTimeoutMs>
</ssl>
...
</vmacore>
- Restart the vpxa process for the changes to take effect.
/etc/init.d/vpxa restart
For ESXi 7.0 U3 and later, refer the below steps:
- From ESXi 7.0 U3, vpxa configuration is stored in special configuration store database. Refer the steps in vSphere ESXi 7.0 U3 VPXA configuration properties (87438) to make changes to vpxa configuration
- By default http and ssl configuration under vmacore properties will look like this.
"vmacore": {
"crashhandling": {},
"debug": {},
"http": {
"default_client_pool_connections_per_server": 300
},
"ssl": {
"do_version_check": false
},
"sys_command": {},
"tcp_keep_alive": {
"client_socket": {},
"server_socket": {}
},
"thread_pool": {
"io_max": 9,
"task_max": 4,
"thread_name_prefix": "vpxa",
"thread_stack_size_kb": 128
}
},
- Add the read_timeout_ms and handshake_timeout_ms under http and ssl sections of vmacore.
- Modified vpxa.cfg file should look like below:
"vmacore": {
"crashhandling": {},
"debug": {},
"http": {
"default_client_pool_connections_per_server": 300,
"read_timeout_ms": 600000
},
"ssl": {
"do_version_check": false,
"handshake_timeout_ms": 600000
},
"sys_command": {},
"tcp_keep_alive": {
"client_socket": {},
"server_socket": {}
},
"thread_pool": {
"io_max": 9,
"task_max": 4,
"thread_name_prefix": "vpxa",
"thread_stack_size_kb": 128
}
},
Feedback
