This article provides information on how to get all services up on the NSX Intelligence node.
Starting HDFS fails.
The User Interface (UI) show services including druid, spark, process and nsx-config are not running.
In the /var/log/hadoop/hadoop-hdfs-namenode-scloud-pace.log file, there is a LogHeaderCorruptException similar to:
2020-04-22 18:14:59,971 WARN org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Log file /data/hadoop/hdfs/current/edits_inprogress_0000000000000503801 has no valid header org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$LogHeaderCorruptException: Unexpected version of the file system log file: 0. Current version = -65. at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.readLogVersion(EditLogFileInputStream.java:380) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:162) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.getVersion(EditLogFileInputStream.java:292) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanEditLog(EditLogFileInputStream.java:347)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware NSX-T Data Center 3.x
Cause
This issue occurs due to corrupted edits log which prevents HDFS namenode to start successfully. This impacts all services to not start as they have dependencies on HDFS.
Resolution
This is a known issue affecting VMware NSX-T Data Center 3.0.0.
Currently, there is no resolution.
Workaround: To work around this issue:
Recover the name node by running this command in the NSX Intelligence appliance:
/opt/apache-hadoop/bin/hadoop namenode -recover
Reply with Y if prompted.
Reboot the NSX Intelligence node.
Additional Information
Please note that this KB applies only to NSX Intelligence versions prior to 3.2 which are offered in OVA format.