NSX Edge Upgrade failed with error: "Edge group upgrade status is failed for group"
search cancel

NSX Edge Upgrade failed with error: "Edge group upgrade status is failed for group"

book

Article ID: 443021

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • NSX Edge upgrade initiated from SDDC failed with the error:
    • Edge group upgrade status is failed for group <Group_UUID>: [Prepare edge upgrade bundle https://<manager_IP>/repository/4.2.3.3.0.25171318/Edge/nub/VMware-NSX-edge-4.2.3.3.0.215171318.nub failed on edge TransportNode <> clientType: Edge, target edge fabric node id <>, return status Download and verify bundle failed with msg: Closing connection 0]
  • Before the Edge upgrade, during the pre-check stage the REPO_SYNC service status had moved to failed state.
  • After resolving the REPO_SYNC issue, as per article: NSX Upgrade Precheck Fails with Repository Error, Edge upgrade failed with the above error.
  • The Edge nodes syslog file reports the following error in the /var/log/syslog file:

YYYY-MM-DDTHH:MM:SS.770Z EdgeNode1NSX 1121 - [nsx@6876 comp="nsx-edge" subcomp="upgrade-agent" tid="1480" level="ERROR" errorCode="MPA50007"] Error downloading nub 'https://<manager_IP>/repository/4.2.3.3.0.25171318/Edge/nub/VMware-NSX-edge-4.2.3.3.0.25171324.nub', output msg: , error msg: * Trying (with httplib) <manager_IP>:443...#012* certificate verification <certificate> from <manager_IP>:443 failed: curl_wrapper failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. Use the --thumbprint or --cacert option.#012* Closing connection 0#012curl_wrapper: (60) curl_wrapper failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. Use the --thumbprint or --cacert option.#012

  • The Edge failed to communicate with the NSX manager and download the upgrade bundle to proceed further with the upgrade.
  • Ping and port connectivity from the Edge node was working as expected.
  • The "get managers" and "get controllers" command provide the details of the NSX manager nodes.
  • The NSX manager's /var/log/upgrade-coordinator/upgrade-coordinator.log file report the following error at the time when the upgrade failed:

YYYY-MM-DDTHH:MM:SS.428Z  INFO task-executor-10-1-workitem-EDGE-<Edge_UUID> Utils 686816 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] getIpv4FromFqdn() invoked with FQDN <manager_IP>.domainfqdn
YYYY-MM-DDTHH:MM:SS.428Z  INFO task-executor-10-1-workitem-EDGE-<Edge_UUID> Utils 686816 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Executing script: /opt/vmware/nsx-common/python/nsx_utils/ip_utils.py -d4 <manager_IP>.domainfqdn
YYYY-MM-DDTHH:MM:SS.559Z  INFO task-executor-10-1-workitem-EDGE-<Edge_UUID> Utils 686816 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] getIpv4FromFqdn(): script returned Ipv4:
...
YYYY-MM-DDTHH:MM:SS.689Z  WARN task-executor-10-1-workitem-EDGE-<Edge_UUID> DnsLookupProviderImpl 686816 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] IPaddressils.getAllIpFromFqdn return [] for FQDN: <manager_IP>.domainfqdn
YYYY-MM-DDTHH:MM:SS.689Z  WARN task-executor-10-1-workitem-EDGE-<Edge_UUID> DnsLookupProviderImpl 686816 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] Fail to get IP address from FQDN <manager_IP>.domainfqdn
YYYY-MM-DDTHH:MM:SS.689Z  WARN task-executor-10-1-workitem-EDGE-<Edge_UUID> UcCertThumbServiceImpl 686816 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] IP-address is empty.

  • The above events confirmed that the DNS resolution for the NSX manager was failing which was leading to the failure of upgrade task.
  • However, nslookup for manager FQDN from the Edge node to the NSX manager is successful.

Environment

VMware NSX

Cause

  • The "opt/vmware/nsx-common/python/nsx_utils/ip_utils.py", script uses "dig" command to perform forward and reverse lookups.
  • The dig command used for DNS resolution by the "ip_utils.py" script was failing.
  • Forward lookup:
    • dig <FQDN of Manager> +short
  • Reverse lookup:
    • dig -x <IP of Manager> +short
  • Packet trace ran while using the dig command, confirmed that the DNS server responded with a FormErr error or ignored the request.

Edge> tcpdump -ni eth0 port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
HH:MM:SS.827184 IP Edge_IP.50448 > DNS_Server_IP.53: 18165+ PTR? <NSX_manager_FQDN>. (44)
HH:MM:SS.827364 IP DNS_Server_IP.53 > Edge_IP.50448: 18165 1/0/0 PTR <NSX_manager_FQDN> (90)
HH:MM:SS.827804 IP Edge_IP.42303 > DNS_Server_IP.53: 45698+ A? <NSX_manager_FQDN> (50)
HH:MM:SS.827971 IP DNS_Server_IP.53 > Edge_IP.42303: 45698 1/0/0 A <NSX_manager_FQDN> (66)
HH:MM:SS.828011 IP Edge_IP.59480 > DNS_Server_IP.53: 7381+ PTR? <NSX_manager_FQDN>. (44)
HH:MM:SS.828099 IP DNS_Server_IP.53 > Edge_IP.59480: 7381 1/0/0 PTR <NSX_manager_FQDN> (90)
HH:MM:SS.828298 IP Edge_IP.34567 > DNS_Server_IP.53: 57654+ A? <NSX_manager_FQDN> (50)
HH:MM:SS.828514 IP DNS_Server_IP.53 > Edge_IP.34567: 57654 1/0/0 A <NSX_manager_FQDN> (66)
HH:MM:SS.828709 IP Edge_IP.34385 > DNS_Server_IP.53: 55680+ [1au] A? <NSX_manager_FQDN>. (78)
HH:MM:SS.828987 IP DNS_Server_IP.53 > Edge_IP.34385: 55680 FormErr- 0/0/1 (78)      <<<< FormErr sent by the DNS server during name resolution

  • FormErr or Format error means DNS server did not understand the request sent by the nodes.
  • Modern versions of dig enable DNS cookies by default. If the DNS servers does not support this feature, it will cause the dig command to fail in resolving the NSX manager FQDN or IP address.
  • Due to this the Edge node was unable to fetch the upgrade bundle from the manage node leading to upgrade task failure.The failed DNS resolution also had caused the failed REPO_SYNC issue.

Resolution

  • To verify if the DNS server is unable to process the cookies attached with the requests, log into the NSX Manager shell as root.
  • Execute a manual DNS query while stripping the DNS Cookie using the +nocookie flag:
    • dig @<DNS_server_IP> <Manager_FQDN> +nocookie
  • If the query successfully returns an IP address without a FormErr, the upstream DNS server's EDNS0 parsing behavior is the root cause.
  • To resolve the issue, patch or upgrade the upstream DNS servers to a supported build that accurately parses EDNS0 extensions and DNS Cookies without throwing a Format Error.
  • Alternatively, reconfigure the NSX nodes to point to a modernized DNS server capable of properly processing standard EDNS0 DNS queries.

Additional Information

For more information on how to add DNS servers on NSX, please refer: Updating DNS server details for NSX-T Manager cluster

Powered by Wolken Software