Corrupted vecs-cli executable impacts Certificate Management and VAMI-based Backups

Products

• VMware vCenter Server

Issue/Introduction

Users may encounter [Errno 8] Exec format error when performing certificate operations or VAMI-based backups on a vCenter. This error typically appears in the following scenarios:

1. VAMI Backup Failure:
   
   • In /var/log/vmware/applmgmt/backup.log, entries similar to the following are found:
     yyyy-mm-ddThh:mm:ss.mss ERROR: Failed to get etcd snapshot for all supervisor clusters: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'Failed to complete backup: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'
2. Service Crashes: 
   • Core certificate services, including vmware-certificateauthority and vmware-certificatemanagement, are in a Stopped state. Manual restart attempts result in an immediate crash or timeout
3. In /var/log/vmware/vmon/vmon.log:
   • Files record specific execution failures: OSError: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'
4. Certificate Manager Utility Fails:
   • Attempting to run the certificate-manager tool results in an OSError:
     File "/usr/lib/vmware-vmca/bin/certificate-manager", line 740, in parse_arguments vecs - Vecsops ()File "/usr/lib/vmware/site-packages/cis/utils.py", line 398, in run_command process = subprocess. Popen (cmd, stdout=subprocess. PIPE,...OSError: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'

5. vecs-cli Binary is 0 Bytes:

   • ls -l /usr/lib/vmware-vmafd/bin/vecs-cli -r-xr-xr-x 1 root root 0 Month Day Year vecs-cli

Environment

• vCenter 7.x
• vCenter 8.x
• vCenter 9.x

Cause

The vecs-cli binary has become 0 bytes in size, which prevents the operating system from executing it. This can be caused by filesystem corruption or manual administrative error (such as mistakenly overwriting the file).

Resolution

To resolve this issue, replace the corrupted vecs-cli executable with a healthy copy from a vCenter Server Appliance (VCSA) of the exact same version and build number.

1. Identify a healthy VCSA running the identical version and build number as the affected server.
   
   
2. Use a utility like WinSCP or SCP to copy /usr/lib/vmware-vmafd/bin/vecs-cli from the healthy vCenter to the /tmp directory of the affected vCenter.
   
   
3. Back up the corrupted file: cp /usr/lib/vmware-vmafd/bin/vecs-cli /tmp/vecs-cli_backup
   
   
4. Use rsync to overwrite the 0-byte file while preserving original permissions: rsync --inplace /tmp/vecs-cli /usr/lib/vmware-vmafd/bin/vecs-cli
   
   
5. Ensure the binary now shows a non-zero size (e.g., approximately 32,368 bytes): ls -l /usr/lib/vmware-vmafd/bin/vecs-cli
   
   
6. Restart all services: service-control --stop --all && service-control --start --all