Removal of inaccessible NFS datastore on hosts fails due to missing credentials for hosts

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

When the SSH credentials for hosts are not present, the removal of inaccessible NFS datastore on those hosts fails.
 

Workflow fails with following exceptions observed in sddc_migration_upgrade.log

2023-04-18T07:43:42.138+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.e.s.o.c.ProcessingTaskSubscriber,pool-5-thread-2] Invoking task FetchInputsForNfsDatastore.PREVALIDATE Description: Fetches input for deleting inaccessible NFS datastore on hosts, Plugin: DriftPlugin, ParamBuilder null, Input map: {credentials=ConfigDrift____0__credentials, inventory=ConfigDrift____0__inventory}, Id: 7f000001-8793-10c3-8187-93510e7a001e ...
2023-04-18T07:43:42.141+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.vcf.migration.util.RestUtils,main] Status Code : [200 OK] for request uri : [http://localhost/lcm/upgrades/7523b4b9-c09e-4ebe-bdf7-f9403740be59] with method : [PATCH]
2023-04-18T07:43:42.142+0000 INFO [vcf_migration,0000000000000000,0000] [c.v.v.migration.helper.UpgradeHelper,main] Updated the upgrade element for upgrade_id 7523b4b9-c09e-4ebe-bdf7-f9403740be59
2023-04-18T07:43:42.241+0000 DEBUG [vcf_migration,0000000000000000,0000] [c.v.e.s.o.c.ProcessingTaskSubscriber,pool-5-thread-2] Invoking task FetchInputsForNfsDatastore. Description: Fetches input for deleting inaccessible NFS datastore on hosts, Plugin: DriftPlugin, ParamBuilder null, Input map: {credentials=ConfigDrift____0__credentials, inventory=ConfigDrift____0__inventory}, Id: 7f000001-8793-10c3-8187-93510e7a001e ...
2023-04-18T07:43:42.357+0000 ERROR [vcf_migration,0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,pool-5-thread-2] [O5R135] FAILED_TO_FIND_ENTITY_CREDENTIAL Failed to find d0b9ee46-5cf4-4c70-913a-71be0f94bbae entity credentials associated with MANAGEMENT Workload Domain in the SDDC Manager Inventory
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to find d0b9ee46-5cf4-4c70-913a-71be0f94bbae entity credentials associated with MANAGEMENT Workload Domain in the SDDC Manager Inventory
 at com.vmware.vcf.migration.actions.FetchInputsForNfsDatastore.lambda$getCredential$3(FetchInputsForNfsDatastore.java:123)
 at java.base/java.util.Optional.orElseThrow(Optional.java:408)
 at com.vmware.vcf.migration.actions.FetchInputsForNfsDatastore.getCredential(FetchInputsForNfsDatastore.java:119)
 at com.vmware.vcf.migration.actions.FetchInputsForNfsDatastore.lambda$execute$1(FetchInputsForNfsDatastore.java:64)
 at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
 at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
 at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)

Environment

VMware Cloud Foundation 5.0

Cause

This issue can arise when the SSH credential for hosts is missing.

platform=# select * from credential where entityid='d0b9ee46-5cf4-4c70-913a-71be0f94bbae';
 id | creationtime | credentialtype | entityid | entitytype | modificationtime | secret | username | serviceid | servicetype | targettype
----+--------------+----------------+----------+------------+------------------+--------+----------+-----------+-------------+------------
(0 rows)

Resolution

To resolve the issue, please follow the below mentioned steps:

1. Decommission the failed hosts from SDDC Manager.

2. Retry the failed workflow and verify if it goes through.