Cloud Proxy Deployment Failure - LCMVSPHERECONFIG1000016 (Truncated TAR archive)

Products

• VMware vCenter Server

Issue/Introduction

When attempting to deploy a new Cloud Proxy, scale-out an existing deployment, or import a proxy across multiple sites using the VCF Operations Fleet Management Appliance, the task fails during the deployment phase.

The UI displays the following generic error:

Error Code: LCMVSPHERECONFIG1000016
Message: IO Exception occurred while performing the operation. Check the logs for more information.
Details: Unexpected ioexception occurred

 

When reviewing the primary lifecycle log via SSH (/var/log/vrlcm/vmware_vrlcm.log), the following explicit system exception is recorded during the DeployOvfTask process:

XXX-XX-XXTXX:XX:XX:XXZ INFO vrlcm[1255] [http-nio-8080-exec-6] [c.v.v.1.r.c.RequestController]
"messageId" : "LCMVSPHERECONFIG1000016",
"message" : "IO Exception occurred while performing the operation. Check the logs for more information.",
"eventId" : "5f8226ef-269e-496b-ae92-09aa2a2f8973",
"retry" : true,
"exceptionMessage" : "Unexpected "exceptionStackTrace" : "java. io. IOException: Truncated TAR archive\n\tat org.apache.commons.compress.archivers.tar.TarArchiveInputStream. read (TarArchiveInputStream. java: 614) \n\tat java.b
ase/java. io. BufferedInputStream. read1 (Unknown Source) \n\tat java. base/java. io.BufferedInputStream. read (Unknown Source) \n\tat com. vmware. vrealize. lcm. drivers. vsphere65.deploy. impl.OvfDeployu
rl.getVMDKFile (OvfDeployUr1. java: 146) \n\tat com. vmware. vrealize. lcm. drivers. vsphere65. deploy. impl.BaseOvfDeploy.getVMDKFileUrl (BaseOvfDeploy. java: 1003) \n\tat com. vmware. vrealize. 1cm.drivers
.vsphere65. deploy. impl. BaseOvfDeploy. importOvf (BaseOvfDeploy. java: 706) \n\tat com. vmware. vrealize. lcm. plugin. core. vsphere. tasks. DeployOvfTask. execute (DeployOvfTask. java: 251) \n\tat com. vmware
.vrealize. lcm.plugin. core. vsphere. tasks. DeployovfTask. retry (DeployovfTask. java: 435) \n\tat com. vmware. vrealize. lcm. automata. core. TaskThread. run (TaskThread. java: 60) \n\tat java.base/java. util.
concurrent. ThreadPoolExecutor. runWorker (Unknown Source) \n\tat java.base/java. util. concurrent. ThreadPoolExecutor$Worker. run (Unknown Source) \n\tat java.base/java. lang. Thread. run (Unknown Sourc
e) \n",
"localizedMessageId" : "VSPHERE_IOEXCEPTION",
"parameters" : null,
"properties" :
"cluster" : "XXXXXXXXXXXXXXXXXXXXXXXXXXX",
"installerLocation" : "http://localhost: 8080/repo/productBinariesRepo/vrops/9.0.2.0/install/cloudproxy.ova",
"storage" : "XXXXXXXXXXXXXXXXXXXXXXXXXX",
"folderName" : "",
"network" : "XXXXXXXXXXXX",
"resourcePool" : ""

-- Retry error cause data : [ (

ioexception occurred.",

 

Environment

VCF 9.0.x
VCF Operations Fleet Management Appliance
Aria Operations / vRealize Operations Cloud Proxy (e.g., version 9.0.2)

Cause

This issue is caused by a severely truncated or corrupted Cloud Proxy OVA binary residing in the internal repository of the Fleet Management appliance.

Because an OVA file is natively a TAR archive, the deployment orchestration engine throws a Truncated TAR archive EOF (End of File) exception when it attempts to stream the unexpectedly short file to vCenter.

In integrated VCF Operations environments, the standalone UI for managing and deleting binaries is locked down. Furthermore, the backend repository stores binaries as alphanumeric hashed blobs (e.g., stripping the .ova extension), making standard file-name searches fail. Deleting the corrupted blob at the OS level will break the internal PostgreSQL database mappings.

Resolution

To resolve this issue without breaking the database mapping or relying on the deprecated standalone UI, you must locate the hidden corrupted blob on the filesystem and overwrite it with a healthy file.

Step 1: Locate the Hidden Binary Blobs
Connect to the VCF Operations Fleet Management appliance via SSH as the root user. First, locate all large files inside the repository blob store to narrow down the suspects.

Command: find /data/vm-config/vmrepo/productBinariesRepo -type f -size +100M -exec ls -lh {} +
Note: A healthy Cloud Proxy OVA is typically between 2.8 GB and 3.1 GB. Note any suspiciously small files (e.g., 1.4 GB or 1.7 GB).

Step 2: Identify the Exact Corrupted Blob
Because the filenames are hashed, read the internal OVF descriptor of each large file to identify which blob specifically belongs to the Cloud Proxy. Run the following loop:

Command: for file in /data/vm-config/vmrepo/productBinariesRepo/XX/XXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXX; do echo -n "Checking $file: "; tar -tf "$file" 2>/dev/null | head -n 1; done
Scan the output for a file returning the Cloud Proxy descriptor (e.g., Operations-Cloud-Proxy-9.0.2.0.xxxx.ovf).
Copy the full, absolute path to this file. (Example: /data/vm-config/vmrepo/productBinariesRepo/XX/XXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXX)

Step 3: Stage a Healthy OVA
Download a fresh, verified copy of the required Aria Operations Cloud Proxy OVA from the official Broadcom Customer Connect portal.
Using a tool like WinSCP or scp, transfer the healthy OVA to the /tmp/ directory of your Fleet Management appliance.
For ease of use, ensure the uploaded file is named cloudproxy.ova within the /tmp/ directory.

Step 4: Overwrite the Corrupted Blob
Execute the following commands to forcefully overwrite the corrupted file and correct the permissions. Replace the <TARGET-FILE-PATH> with the exact path you identified in Step 2.

Overwrite the file:
Command 1: cp /tmp/cloudproxy.ova /data/vm-config/vmrepo/productBinariesRepo/XX/XXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXX

Apply proper read permissions:
Command 2: chmod 644 /data/vm-config/vmrepo/productBinariesRepo/XX/XXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXX

Reassign system ownership:
Command 3: chown root:root /data/vm-config/vmrepo/productBinariesRepo/XX/XXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXX

Step 5: Retry the Deployment
Because the underlying file has been replaced while retaining the original filename and path, the internal database remains intact.

Return to your management portal and re-trigger the Cloud Proxy deployment or scale-out task.

The deployment will now process the complete archive and successfully stream the OVF to the target vCenter.