"FileNotFoundError: [Errno 2] No such file or directory: '/storage/seat/cis-export-folder/vmafd/data/vmdird'", Convergence fails around 42% during VMAFD firstboot

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Converge fails at 42% with error "FileNotFoundError: [Errno 2] No such file or directory: '/storage/seat/cis-export-folder/vmafd/data/vmdird'"
• In log /var/log/vmware/converge/conver.log, you see entries similar to :

2020-03-24T21:59:20.715Z ERROR converge Failed to run white listed firstboot scripts. Error: {
    "resolution": null,
    "problemId": null,
    "componentKey": null,
    "detail": [
        {
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['run-firstboot-scripts', '--action', 'firstboot', '--subaction', 'firstboot', '--fbWhiteList', 'visl-support-firstboot,vmafd-firstboot']\nStderr: "
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['run-firstboot-scripts', '--action', 'firstboot', '--subaction', 'firstboot', '--fbWhiteList', 'visl-support-firstboot,vmafd-firstboot']\nStderr: '"
        }
    ]
}

• In log /var/log/firstboot/vmafd-firstboot.py_<PID>_stdout.log, you see entries similar to:

2020-03-24T21:59:20.660Z  Importing directory data. This will take a few minutes.
2020-03-24T21:59:20.666Z  Getting value for install-parameter: upgrade.import.directory
2020-03-24T21:59:20.666Z  VMAFD Firstboot failed

 

• In log /var/log/firstboot/vmafd-firstboot.py_<PID>_stderr.log, you see entries similar to:

2020-03-24T21:59:20.667Z  Exception: Traceback (most recent call last):
  File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 177, in main
    controller.firstboot()
  File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 51, in firstboot
    self.start()
  File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 67, in start
    service.start()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmdirUpgrade.py", line 586, in start
    self.start_upgrade()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmdirUpgrade.py", line 451, in start_upgrade
    self.import_data_files()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmdirUpgrade.py", line 389, in import_data_files
    shutil.copytree(importDir, self.data_path)
  File "/usr/lib/python3.5/shutil.py", line 309, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/storage/seat/cis-export-folder/vmafd/data/vmdird'

Environment

VMware vCenter Server 6.7.x

Cause

This issue is caused due to stale install parameter file '/etc/vmware/install-defaults/upgrade.import.directory' which was not cleaned up during previous vCenter Server upgrade process. VMAFD firstboot during the converge process will proceed with upgrade workflow due to the existence of this file and tries to read the exported config files.

Resolution

1. Revert the failed vCenter Server and PSC to a snapshot which was taken before trying the Convergence. All PSCs needs to be reverted to the snapshot in case vCenter Servers are in linked mode or part of same SSO domain. This step is important as the vCenter Services will not start after a failed convergence process. If no snapshots, follow Step 4 to 15 on KB https://kb.vmware.com/s/article/68159 to recover from the failed situation

2. Delete the file /etc/vmware/install-defaults/upgrade.import.directory, this is stale install parameter file which was not cleaned up during vCenter Server upgrade process

rm /etc/vmware/install-defaults/upgrade.import.directory

3.Retry the Convergence.

Additional Information

Impact/Risks:
Please take cold snapshot of all PSCs and vCenter Servers in the same SSO Domain before trying Converge operation.