Starting HDFS fails on the NSX Intelligence node

Products

• VMware NSX-T Data Center

Issue/Introduction

This article provides information on how to get all services up on the NSX Intelligence node.

• Starting HDFS fails.
• The User Interface (UI) show services including druid, spark, process and nsx-config are not running.
• In the /var/log/hadoop/hadoop-hdfs-namenode-scloud-pace.log file, there is a LogHeaderCorruptException similar to:
  
  2020-04-22 18:14:59,971 WARN org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Log file /data/hadoop/hdfs/current/edits_inprogress_0000000000000503801 has no valid header
  org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$LogHeaderCorruptException: Unexpected version of the file system log file: 0. Current version = -65.
          at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.readLogVersion(EditLogFileInputStream.java:380)
          at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:162)
          at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.getVersion(EditLogFileInputStream.java:292)
          at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanEditLog(EditLogFileInputStream.java:347)
          
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.   

Environment

VMware NSX-T Data Center 3.x

Cause

This issue occurs due to corrupted edits log which prevents HDFS namenode to start successfully. This impacts all services to not start as they have dependencies on HDFS.

Resolution

This is a known issue affecting VMware NSX-T Data Center 3.0.0.

Currently, there is no resolution.

Workaround:
To work around this issue:

1. Recover the name node by running this command in the NSX Intelligence appliance:
   
   /opt/apache-hadoop/bin/hadoop namenode -recover
    
2. Reply with Y if prompted.
3. Reboot the NSX Intelligence node.

Additional Information

Please note that this KB applies only to NSX Intelligence versions prior to 3.2 which are offered in OVA format.