Error 'LCMVSPHERECONFIG1000016' and 'java.io.IOException: Error writing request body to server' during VCF Operations for Networks deployment with Fleet Manager

Products

• VCF Operations for Networks

Issue/Introduction

When attempting to deploy or expand a VCF Operations for Networks cluster via Fleet Manager, the operation fails for one or more nodes. The task terminates during the OVF deployment phase.

The user interface reports the error code:

Error Code: LCMVSPHERECONFIG1000016
IO Exception occurred while performing the operation. Check the logs for more information.
Unexpected ioexception occurred.

java.io.IOException: Error writing request body to server    at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(Unknown Source)    at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(Unknown Source)    at java.base/java.io.BufferedOutputStream.write(Unknown Source)    at com.vmware.vrealize.lcm.drivers.vsphere65.deploy.impl.OvfDeployUrl.getVMDKFile(OvfDeployUrl.java:138)    at com.vmware.vrealize.lcm.drivers.vsphere65.deploy.impl.BaseOvfDeploy.getVMDKFileUrl(BaseOvfDeploy.java:1003)    at com.vmware.vrealize.lcm.drivers.vsphere65.deploy.impl.BaseOvfDeploy.importOvf(BaseOvfDeploy.java:706)    at com.vmware.vrealize.lcm.plugin.core.vsphere.tasks.DeployOvfTask.execute(DeployOvfTask.java:251)    at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62)    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)    at java.base/java.lang.Thread.run(Unknown Source)


The vmware_vrlcm-#.log (located at /var/log/vrlcm/) on the Fleet Manager appliance contains the following error pattern:

<timedatestamp> ERROR vrlcm[1311] [pool-3-thread-18] [c.v.v.l.d.v.d.i.BaseOvfDeploy]  -- Exception occured while uploading ova files to vCenter. Aborting the lease. Error:{}
java.io.IOException: Error writing request body to server

 

NOTE:  VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI). Fleet Manager was formerly known as vRealize Suite Lifecycle Manager (vRSLCM or, commonly, LCM)

Environment

VCF Operations for Networks

VCF Fleet Manager

VCF Cloud Foundation (VCF)

Cause

This error is caused by network issues between the Fleet Manager appliance, the vCenter and/or the target ESXi host where the OVF is being uploaded.

The java.io.IOException: Error writing request body to server indicates that the TCP connection was established but terminated prematurely during the data transmission of the OVF/VMDK payload.

Possible causes include:

• Asymmetric routing between the Fleet Manager appliance, the vCenter and/or the target ESXi host where the OVF is being uploaded

• MTU inconsistencies along the data paths causing packet drops for large frames by devices configured for smaller MTUs

• Firewall rules severing long-lived HTTP/HTTPS connections

Resolution

To resolve this issue, validate the data paths between the Fleet Manager appliance, the vCenter and/or the target ESXi host where the OVF is being uploaded before retrying the deployment.

1. Audit Network Routing: Verify the routing tables for the Fleet Manager appliance and the upstream gateway. Ensure a valid, symmetric path exists between the Fleet Manager appliance, the vCenter and/or the target ESXi host where the OVF is being uploaded.

2. Retry Deployment: Once the data paths have been validated as above, log in to the Fleet Manager interface and retry the node deployment or cluster expansion task.