TKGI Cluster upgrade is failing on master : 1 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm.

Products

• VMware NSX
• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

• TKGI Master node is present on NSX overlay network
• Ping between Master node and NSX Manager fails
• Traceroute between Master node and NSX Manager fails on T0
• ICMP traffic is reaching the NSX Manager, return traffic is being dropped on the T0
• T0 Logical Router may show duplicate routes back to different T1s

T0>get logical-routers
...
t1n> * 10.10.10.10/32 [3/0] via 100.64.66.1, downlink-123
t1n> * 10.10.10.10/32 [3/0] via 100.64.67.2, downlink-456
...

• SNAT rule on separate T1s are advertising the same SNAT IP to T0

Environment

VMware NSX
VMware NSX-T Data Center
VMware Tanzu Kubernetes Grid Integrated

Resolution

To resolve the issue the problematic rule can be disabled. The TKGI SNAT rules cannot be edited within the NSX User Interface. To adjust the rule the API must be used.

Get SNAT - Get rule

Put SNAT - Put rule

Include the body of the GET output in a PUT call. Change enabled field to "false".

Note:  "X-Allow-Overwrite":"true" must be included in the header for the PUT call.

Additional Information

Note: Before making any changes to NAT rules, verify that the rule is not in use.