VCF on VxRail upgrade from 3.7.1 to 3.7.2 fails on Configuration Drift.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:
While completing the configuration drift upgrade from 3.7.1 to 3.7.2, the step fails.

In the log /var/log/vmware/vcf/lcm/upgrades/<upgrade-id>/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade.log, similar error messages are seen:
[threadPoolExecutor-4] ERROR c.v.e.s.o.model.error.ErrorFactory - [5NP5UP] INTERNAL_SERVER_ERROR Invocation of prefix '' part of task AddVcenterToKnownHosts in plugin MigrationPlugin failed with exception.
com.vmware.evo.sddc.common.core.error.InternalServerErrorException: Invocation of prefix '' part of task AddVcenterToKnownHosts in plugin MigrationPlugin failed with exception.
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:362)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:407)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:96)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
        at reactor.bus.EventBus$3.accept(EventBus.java:317)
        at reactor.bus.EventBus$3.accept(EventBus.java:310)
        at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
        at reactor.bus.EventBus.accept(EventBus.java:591)
        at reactor.bus.EventBus.accept(EventBus.java:63)
        at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
        at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to 172.18.33.30
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:144)
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:87)


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation on VxRail 3.7.x

Cause

This issue is seen when all Workload Domain vCenters do not share the same root password. The upgrade process finds the first vCenter credentials and uses this for each subsequent SSH attempt to the other vCenters. This will fail on a vCenter that has a different root password.

Resolution

This is a known issue with no resolution at this time. 

To workaround this issue, please download the attached update_workload_domains_vcenter_password.py script and then follow the steps below:

1. Using WinSCP, or another file transfer utility, copy the update_workload_domains_vcenter_password.py script to the SDDC Manager VM directory /opt/vmware/sddc-support/dependency/.
2. SSH to the SDDC manager and log in as vcf user.
3. Navigate to the /home/vcf/ directory and execute the script by running the below command:

python update_workload_domains_vcenter_password.py

4. The script should complete with a message "Successfully updated vCenter password for workload domains". Once this is shown the workflow can be restarted.