VMware Blockchain replica nodes are unresponsive

Products

• VMware Blockchain

Issue/Introduction

Symptoms:

• You are unable to login to VMware Blockchain (VMBC) replica nodes as the root or vmbc user.
• You are unable to power off VMBC replica node virtual machines. You see an error similar to the following:

Error : A general system error occurred: Invalid fault

• You see an error similar to the following if you try to reset the root password:

Authentication Token Manipulation Error ; Password unchanged

 

Cause

This issue can occur when when the root partition on the VMBC replica node is full.

Resolution

1. Check for free disk space on the affected node with the the df -h command.
2. Run the following command to check filesystem usage (sorted by directory):

du -h -d 2 -x / 2>@1 | sort -rh | head

3. Run the following command to shrink the systemd journal:

journalctl --vacuum-size=50M

4. Run the following command to find log files within containers without accessing containers sorting by largest:

find /var/lib/docker/containers -name \*.log* -ls | sort -r -n -k7

5. Ensure that the endpoints for the fluentd and telegraf containers are are updated correctly in their respective configuration files.
6. A command similar to the following can be used to find and remove large log files in bulk from the current working directory: 

   for i in $(ls -1 | grep -i "\.log$"); do rm $i; done; 
   
   
   

Additional Information

Impact/Risks:
This issue will severely impact the functionality of the VMBC replica node. You may also see VMBC upgrades fail because of this issue.