Corrupted vecs-cli Executable Impacts Certificate Mangement and VAMI based backups
search cancel

Corrupted vecs-cli Executable Impacts Certificate Mangement and VAMI based backups

book

Article ID: 410607

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

[Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'

This error may be seen in one more locations (logs, tool output, etc.).

VAMI Backups Failing:

 

In /var/log/vmware/applmgmt/backup.log, log messages similar to the following are found:

yyyy-mm-ddThh:mm:ss.mss [20250904-151054-24322831] [ComponentScriptsBackup:PID-4027345] [Log::run:Log.py:64] ERROR: Failed to get etcd snapshot for all supervisor clusters: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'Failed to complete backup: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'
 

Certificate Manager Utility Fails:

 

root@vcenter [ /usr/11b/vmware-vmca/bin ]# ./certificate-manager

*** Welcome to the vSphere 8.0 Certificate Manager ***

 -- Select Operation --

 1. Replace Machine SSL certificate with Custom Certificate

 2. Replace VHCA Root certificate with Custom Signing
   Certificate and replace all Certificates

 3. Replace Machine SSL certificate with VMCA Certificate

 4. Regenerate a new VMCA Root Certificate and replace all certificates

 5. Replace Solution user certificates with Custom Certificate
   NOTE: Solution user certs will be deprecated in a future release of vCenter. Refer to release notes for more details.

 6. Replace Solution user certificates with VMCA certificates

 7. Revert last performed operation by re-publishing old certificates

 8. Reset all Certificates

 Note : Use Ctrl-D to exit.
 Option[1 to 8]: 4
 Do you wish to generate all certificates using configuration file : Option[Y/N] ? : n

 Please provide valid SSO and VC privileged user credential to perform certificate operations.
 Enter username [[email protected]] :
 Enter password:

 Traceback (most recent call last) :
 File "/usr/lib/vmware-vmca/bin/./certificate-manager", line 766, in main parse arguments ()
 File "/usr/lib/vmware-vmca/bin/./certificate-manager", line 740, in parse_arguments vecs - Vecsops ()
 File "/usr/lib/vmware/site-packages/cis/certificateManagerOps.py", line 817, in init stores = self.list_stores ()
 File "/usr/lib/vmware/site-packages/cis/certificateManagerOps.py", line 937, in list_stores (code, result, err) = run_command (cmd, None, True)
 File "/uar/l1b/vmware/site-packages/cis/utils.py", line 398, in run_command process = subprocess. Popen (cmd, stdout=subprocess. PIPE,
 File "/usr/lib/python3.10/subprocess.py", line 971, in init self. execute_child(args, executable, preexec_fn, close_fds,
 File "/usr/lib/python3.10/subprocess.py", line 1751, in _execute_child self. posix_spawn (args, executable, env, restore_signals,
 File "/usr/lib/python3.10/subprocess.py", line 1696, in posix_spawn self.pid - os.posix_spawn (executable, args, env, ** kwargs) 

OSError: [Errno 8] Exec format error: '/usr/lib/vmware-vmafd/bin/vecs-cli'

 Certificate Manager tool do not support vCenter HA systems

 

 

vecs-cli Binary is 0 Bytes in Size:

 

root@vcenter [ ~ ]# ls -l /usr/lib/vmware-vmafd/bin/
total 224
-r-xr-xr-x 1 root root  19816 Month  Day  Year cdc-cli
-r-xr-xr-x 1 root root 106824 Month  Day  Year dir-cli
-r-xr-xr-x 1 root root  15816 Month  Day  Year domainjoin
-r-xr-xr-x 1 root root  15688 Month  Day  Year vdcpromo
-r-xr-xr-x 1 root root      0 Month  Day  Year vecs-cli <-------------------
-r-xr-xr-x 1 root root  32344 Month  Day  Year vmafd-cli

 

Environment

  • vCenter 7.x
  • vCenter 8.x
  • vCenter 9.x

Cause

The root cause of the binary becoming 0 bytes in size could be either filesystem corruption, or a manual mistake (like mistakenly overwriting the file with an empty string).

Resolution

To resolve the issue, replace the non-working vecs-cli executable with a healthy copy obtained from a working vCenter of the same version and build number.

Steps:

  1. Identify a healthy VCSA running the exact same version and build number as the affected vCenter.

  2. Using WinSCP (or alternative program), copy the functional /usr/lib/vmware-vmafd/bin/vecs-cli file from the healthy vCenter to the affected vCenter.  For example, copy the executable to /tmp/vecs-cli on the unhealthy vCenter.

  3. On the affected vCenter server, backup the existing executable:

    1. cp /usr/lib/vmware-vmafd/bin/vecs-cli /tmp/vecs-cli_backup
  4. Overwrite the existing 0 byte executable while preserving permissions/file properties: 
    1. rsync --inplace /tmp/vecs-cli /usr/lib/vmware-vmafd/bin/vecs-cli
  5. Validate the executable is the correct size: 
    1. root@vcenter [ ~ ]# ls -l /usr/lib/vmware-vmafd/bin/
      total 224
      -r-xr-xr-x 1 root root  19816 Month  Day  Year cdc-cli
      -r-xr-xr-x 1 root root 106824 Month  Day  Year dir-cli
      -r-xr-xr-x 1 root root  15816 Month  Day  Year domainjoin
      -r-xr-xr-x 1 root root  15688 Month  Day  Year vdcpromo
      -r-xr-xr-x 1 root root  32368 Month  Day  Year vecs-cli <------------
      -r-xr-xr-x 1 root root  32344 Month  Day  Year vmafd-cli
  6. Restart services on the vCenter server: 
    1. service-control --stop --all && service-control --start --all
Powered by Wolken Software