NSX Edge node failure due to infrastructure availability or storage latency
search cancel

NSX Edge node failure due to infrastructure availability or storage latency

book

Article ID: 443311

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • You may observe the NSX Edge node transitioning to an unhealthy state.
  • Log lines similar to the below are encountered on the NSX Edge node in /var/log/syslog
    Sample log in /var/log/syslog
    NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-fsm" level="INFO"] HA state Active, processing event Datapath Disconnected reason DP Disconnected
    NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-fsm" level="INFO"] ha_transition_action(Datapath Disconnected, DP Disconnected, 0x1, Offline)
    NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-fsm" level="WARN"] Node HA state transition from Active to Offline, event Datapath Disconnected, reason DP Disconnected
  • Log lines similar to the below are encountered on the NSX Edge node in /var/log/syslog
    Sample log in /var/log/syslog
    NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="ha-cluster" level="WARN" eventId="vmwNSXClusterNodeStatus"] {"event_state":4,"event_external_reason":"Edge node status changed: Up -> Down , reason: HA not in Active state","event_src_comp_id":"########-####-####-####-############","event_sources":{"id":"########-####-####-####-############"}}
  • Monitoring the affected node may reveal a high load average alongside numerous processes in an uninterruptible sleep (D) state. Despite the high load average, the CPU idle (id) percentage remains high, indicating the processor is waiting for I/O rather than executing tasks.
    Sample output of get processes
    top - ##:##:## up ## days, # user,  load average: ##.##, ##.##, ##.##
    Tasks: ### total,   # running, ### sleeping,   # stopped,   # zombie
    %Cpu(s):  #.# us,  #.# sy,  #.# ni, ##.# id,  #.# wa,  #.# hi,  #.# si,  #.# st
        PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
       #### root      20   0     ###    ###    ### S   #.#   #.#     ##:## dp-fp:0
       #### root      19   0     ###    ###    ### D   #.#   #.#     ##:## /lib/systemd/systemd-journald
       #### root      19   0     ###    ###    ### D   #.#   #.#     ##:## /lib/systemd/systemd-journald
       #### root      19   0     ###    ###    ### D   #.#   #.#     ##:## /lib/systemd/systemd-journald
  • Log lines similar to the below are encountered on the NSX Edge node in /var/log/kern.log

    Sample log in /var/log/kern.log
    kernel - - - [###.###] Out of memory: Killed process ### (###) total-vm:###kB, anon-rss:###kB, file-rss:###kB, shmem-rss:###kB, UID:### pgtables:###kB oom_score_adj:###

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

The logs of the NSX Edge node show evidence of D processes piling-up, steadily increasing the CPU load average until the kernel started killing processes due to out of memory.
This behaviour is typically due to storage issues, or other unavailability issues, at the ESXi host layer.

Resolution

This is a condition that may occur in a VMware NSX environment.

Investigate the underlying vSphere and storage infrastructures.

Powered by Wolken Software