"Failure Domain Down" and "Password Expiry" alarms are reported for nodes.

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• "Failure Domain Down" alarm is reported.

• The alarm retriggers after being manually resolved.
• No traffic impact is seen.
• Errors similar to the following example can be seen in /var/log/syslog on the affected NSX Edge node:
  

  2023-08-16T07:51:41.958Z mynsxedge01 NSX 38252 - [nsx@6876 comp="nsx-edge" subcomp="python" username="root" level="ERROR" errorCode="('CLI110',)"] Failed to get Manager connection status: sudo: Account or password is expired, reset your password and try again#012sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper#012sudo: unable to change expired password: Authentication token manipulation error
  2023-08-16T07:53:47.876Z mynsxedge01 NSX 3315 - [nsx@6876 comp="nsx-edge" subcomp="node-mgmt" username="root" level="CRITICAL" eventFeatureName="edge_health" eventType="failure_domain_down" eventSev="critical" eventState="On" entId="00000000-####-####-####-444444444444"] All members of failure domain 00000000-####-####-####-444444444444 are down.

• In /var/log/nvpapi/api_server.log:
  

  2023-08-16T07:51:37.824Z nsx_monitoring.clientlibrary.event_source CRITICAL All members of failure domain 00000000-####-####-####-444444444444 are down.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

• The NSX Manager is unable to retrieve the correct information from the NSX Edge node due to the expiration of the root password.
• The password expiration could also be verified from the NSX UI alarms page.
• Due to a potential race condition in versions before 4.1.2, customers may not receive the expected password expiration alarms; instead, only the Failure Domain Down alarm may be triggered.
• The Failure Domain Down alarm may appear if the root password has expired. Without a valid root password, certain information from the NSX Manager cannot be retrieved, which can trigger this alarm.

Resolution

• NSX version starting 4.1.2 - "Failure domain"  and "password expiry" alarms are expected in alarm section when password expired.
  • If a password is changed before expiry, the failure domain alarm will not appear.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Reset or renew the root password on the affected NSX Edge node using one of the following methods:

• Method A:
  1. From the VM console or SSH on the NSX Edge node.
  2. Authenticate with the admin user.
  3. Update the root password:
     mynsxedge01> set user root password
Current password: <old_password>
New password: <new_password>
Retype new password: <new_password>

• Method B:
  1. From the VM console or SSH on the NSX Edge node.
  2. Attempt to authenticate with the root user.
  3. Usually, login prompt asks you to change password as follows:
     root@mynsxedge01's password: <old_password>
You are required to change your password immediately (password expired)
Changing password for root.
Current password: <old_password>
New password: <new_password>
Retype new password: <new_password>

Additionally, if desired, you may change the password expiration:

1. Connect to the NSX Edge node as admin via console or SSH.
2. Clear the password expiration for user root:
   clear user root password-expiration
3. Query the password expiration for user root:
   get user root password-expiration

If the current, expired root password is not known, the following documents provide steps for utilizing the GRUB menu to trigger a password reset operation. 

Resetting the Passwords of an Appliance for NSX-T 3.2

Resetting the Passwords of an Appliance for NSX 4.1

Additional Information

Failure domain down alarm