VCF Cluster Delete Failure: ""FAILED_TO_FETCH_VALIDATE_NSXT_SERVICE_DEPLOYMENTS_INPUT" or "UNABLE_TO_GET_INPUT_FOR_NSXT

Products

VMware SDDC Manager VMware Cloud Foundation 5.x VMware Cloud Foundation

Issue/Introduction

When attempting to delete/remove a WLD Cluster via the SDDC Manager UI, it may fail with: "FAILED_TO_FETCH_VALIDATE_NSXT_SERVICE_DEPLOYMENTS_INPUT".

You may observe an error in the SDDC Manager - /var/log/vmware/vcf/domainmanager/domainmanager.log:

yyyy-mm-ddThh:mm:ss ERROR [vcf_dm,xxxxxxxxxxxxxxxxxxxxxxx,c3b9] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20]  [H2K7A6] FAILED_TO_FETCH_VALIDATE_NSXT_SERVICE_DEPLOYMENTS_INPUT Failed to fetch validate NSX service deployments input data
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to fetch validate NSX service deployments input data

........................................................

Caused by: java.lang.IllegalStateException: Duplicate key null (attempted merging values [xxxxxxxxxxxx])

Found in domain manager log

2024-10-08T17:47:29.675+0000 WARN [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.v.c.h.i.HttpProtocolBindingBase,dm-exec-19] Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
2024-10-08T17:47:29.749+0000 INFO [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-19] Successfully logged in to https://vcf-shr-vc.utility.pge.com:443/sdk
2024-10-08T17:47:29.751+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-19] Security config retrieved {"fipsMode":false}
2024-10-08T17:47:29.752+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-19] Creating ApiClient to https://vcf-shr-nsx.utility.pge.com:443 with username admin
2024-10-08T17:47:29.752+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-19] Created ApiClient connection to: vcf-shr-nsx.utility.pge.com
2024-10-08T17:47:30.058+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-19] NSX Version: 4.1.2.3.0.23382408, NSX Version with policy baseline: 4.1.1.0.0-0
2024-10-08T17:47:30.058+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-19] Overriding Mp handlers with policy handlers
2024-10-08T17:47:30.178+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.f.p.n.p.a.PrepareSubClusterCreationInputAction,dm-exec-19] The cluster is strecthed and network profile is not defined in the spec.Hence adding hosts: [itvcfrcshresx53.utility.pge.com, itvcfrcshresx54.utility.pge.com] to AZ2
subcluster
2024-10-08T17:47:30.276+0000 DEBUG [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.n.s.c.c.ApiConnection,dm-exec-19] Closed ApiClient connection.
2024-10-08T17:47:30.282+0000 ERROR [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.v.c.f.p.n.p.a.PrepareSubClusterCreationInputAction,dm-exec-19] UNABLE_TO_GET_INPUT_FOR_NSXT_CONFIGURATION
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

...............

2024-10-08T17:47:30.291+0000 ERROR [vcf_dm,67057031eb2ee14d789e56a7333d02a7,1fa3] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-19] [3EP880] UNABLE_TO_GET_INPUT_FOR_NSXT_CONFIGURATION Unable to get input for NSX configuration
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to get input for NSX configuration

Environment

VMware Cloud Foundation 5.x

Cause

The reported error "FAILED_TO_FETCH_VALIDATE_NSXT_SERVICE_DEPLOYMENTS_INPUT" comes from "FetchValidateNsxtServiceDeploymentsInputAction" which is NSX specific action. This issue happens with Policy APIs used in SUB-TNP related implementation in the "Remove Cluster" workflow. i.e. The TransportZoneProfiles contain global transport zone paths.

This happens with environments that have been upgraded from 4.x to 5.x versions of SDDC manager with stretched vSAN clusters.

Resolution

Take snapshot of SDDC Manager VM
Set TNP to false in SDDC Manager properties
1. SSH into the SDDC Manager via the vcf user and su to root.
2. Either edit or create the /home/vcf/feature.properties file and add/edit the following line to false:
```
vi /home/vcf/feature.properties
```
3. Press i for insert mode and add the below line
```
feature.vcf.subtnp=false
```
4. Save and exit
```
esc > :wq!
```
Turn OFF policy APIs on VFC 5.1 environment
1. Update file /etc/vmware/vcf/domainmanager/application-prod.properties with following entry:
```
vi /etc/vmware/vcf/domainmanager/application-prod.properties
```
2. Press i for insert mode
3. add the below line
```
nsxt.base.version.policy=4.2.0.0.0-0
```
4. Save and exit
```
esc > :wq!
```
5. Restart domainmanager by running following command:
```
systemctl restart domainmanager
```
6. Check that domainmanager is active by running following command:
```
systemctl status domainmanager
```
Perform add host(s) operation on SDDC Manager (UI / API)
Turn ON policy APIs on VFC 5.1 environment:
1. Remove following entry from file /etc/vmware/vcf/domainmanager/application-prod.properties:
```
vi /etc/vmware/vcf/domainmanager/application-prod.properties
```
2. delete the below line
```
nsxt.base.version.policy=4.2.0.0.0-0
```
3. Save and exit
```
esc > :wq!
```
4. Restart domain manager
```
systemctl restart domainmanager
```
5. Check that service is active
```
systemctl status domainmanager
```