NSX-T Upgrade failed at task "NSX-T TRANSPORT NODE POSTCHECK STAGE" due to underlying DNS Issues

Products

• VMware Cloud Foundation

Issue/Introduction

NSX-T upgrade on host cluster fails in "NSX-T TRANSPORT NODE POSTCHECK STAGE" with the following errors:

• Check LCM Debug log(Default location: /var/log/vmware/vcf/lcm/lcm-debug.log) for the following text: ‘Temporary failure in name resolution’

• The same error object will also contain this  text: “methodName”:“getUpgradeUnitGroupAggregateInfo”

ERROR[vcf_lcm,ac1b5eda22e9f199,3ce6,upgradeId=2f4c81d9-364c-4104-be3c-a7fe7522a12b,resourceType=NSX
_T_PARALLEL_CLUSTER,resourceId=nsxt-vip.domain.local:_ParallelClusterUpgradeEl
ement,bundleElementId=b4f48dca-4680-48e6-b1fe-8cddcc302d35] [c.v.e.s.l.p.i.n.s.NsxtHostClusterParall
elUpgradeStageRunner,Async-30] Caught exception {"detailMessage":"vum-wld1-nsxt-vip.w2-qes-001.rainp
ole.local: Temporary failure in name resolution","stackTrace":[{"classLoaderName":"app","declaringCl
ass":"com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport","methodName":"exe
cute","fileName":"ApacheClientRestTransport.java","lineNumber":81,"format":1},{"classLoaderName":"ap
p","declaringClass":"com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$Def
aultRequestExecutor","methodName":"execute","fileName":"DefaultRequestExecutorFactory.java","lineNum
ber":45,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware.vapi.internal.protocol.cli
ent.rest.RestClientApiProvider","methodName":"invoke","fileName":"RestClientApiProvider.java","lineN
umber":67,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware.vapi.internal.bindings.S
tub","methodName":"invoke","fileName":"Stub.java","lineNumber":241,"format":1},{"classLoaderName":"a
pp","declaringClass":"com.vmware.vapi.internal.bindings.Stub","methodName":"invokeMethodAsync","file
Name":"Stub.java","lineNumber":191,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware
.vapi.internal.bindings.Stub","methodName":"invokeMethod","fileName":"Stub.java","lineNumber":137,"f
ormat":1},{"classLoaderName":"app","declaringClass":"com.vmware.nsx.upgrade.upgrade_unit_groups.Aggr
egateInfoStub","methodName":"list","fileName":"AggregateInfoStub.java","lineNumber":54,"format":1},{
"classLoaderName":"app","declaringClass":"com.vmware.nsx.upgrade.upgrade_unit_groups.AggregateInfoSt
ub","methodName":"list","fileName":"AggregateInfoStub.java","lineNumber":37,"format":1},{"classLoade
rName":"app","declaringClass":"com.vmware.vcf.common.nsxt.sdk.client.connection.UpgradeUnitGroupOper
ations","methodName":"getUpgradeUnitGroupAggregateInfo","fileName":"UpgradeUnitGroupOperations.java"
,"lineNumber":198,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware.evo.sddc.lcm.pri
mitive.impl.nsxt.service.NsxtHostClusterParallelUpgradeStageRunner","methodName":"performPostcheck",
"fileName":"NsxtHostClusterParallelUpgradeStageRunner.java","lineNumber":222,"format":1},{"classLoad
erName":"app","declaringClass":"com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtHostClusterP
arallelUpgradeStageRunner","methodName":"performHostTypeUpgradePostcheck","fileName":"NsxtHostCluste
rParallelUpgradeStageRunner.java","lineNumber":137,"format":1},{"classLoaderName":"app","declaringCl
ass":"com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtHostClusterParallelUpgradeStageRunner"
,"methodName":"doUpgradeStage","fileName":"NsxtHostClusterParallelUpgradeStageRunner.java","lineNumb
er":106,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware.evo.sddc.lcm.primitive.imp
l.nsxt.NsxtParallelClusterPrimitiveImpl","methodName":"runUpgrade","fileName":"NsxtParallelClusterPr
imitiveImpl.java","lineNumber":431,"format":1},{"classLoaderName":"app","declaringClass":"com.vmware
.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl","methodName":"postUpgrade","file
Name":"NsxtParallelClusterPrimitiveImpl.java","lineNumber":187,"format":1},{"classLoaderName":"app",
…
} while performing stage NSX_T_HOST_TYPE_POSTCHECK
_STAGE. Returning status NSXT_HOST_CLUSTER_UPGRADE_FAILED_POSTCHECK

 

Environment

VMware Cloud Foundation 4.5

VMware NSX

Cause

NSX-T Hostname resolution issues render the sddc manager being unable to communicate with NSX-T manager, thereby resulting in the sddc manager marking the NSXT Host Upgrade as failed in the postcheck stage.

Resolution

Currently there is no resolution to the issue.

Workaround:

To workaround the issue, please follow the below mentioned steps:

• Ssh into SDDC manager

• Verify DNS server connection using nslookup <nsx-t manager FQDN/IPs>for nsxt manager ips and host ips

Sample output after using nslookup:
root@sddc-manager [ ~ ]# nslookup nsx-mgmt-1.vrack.vsphere.local
Server:         10.0.0.250
Address:        10.0.0.250#53

Name:   nsx-mgmt-1.vrack.vsphere.local
Address: 10.0.0.31

• If any errors are reported, resolve them before proceeding.

• Retry the upgrade from the SDDC Manager UI