Commands from ssh session on ESXi host failed with No space left on device error
search cancel

Commands from ssh session on ESXi host failed with No space left on device error

book

Article ID: 427365

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • This issue is only happened in VxRail environment.

  • Commands from ssh session on ESXi host failed with 'No space left on device error'.

  • This issue is only happened on ssh session.

  • If you are using console like using iDRAC, the command result shows well.

    [Example] From ssh session on ESXi host

    [root@localhost:~] passwd
    Traceback (most recent call last):
      File "/bin/passwd", line 56, in <module>
        sys.exit(main())
      File "/bin/passwd", line 48, in main
        retCode = subprocess.call(call_args);
      File "/lib64/python3.11/subprocess.py", line 389, in call
      File "/lib64/python3.11/subprocess.py", line 1026, in __init__
      File "/lib64/python3.11/subprocess.py", line 1885, in _execute_child
    OSError: [Errno 28] No space left on device

    [root@localhost:~] df
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libcore-storage.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libcore-storage.so : libboost_filesystem-mt-x64.so.1.78.0: failed to map segment from shared object
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libdaemon.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libdaemon.so : libdvs.so: failed to map segment from shared object
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libdevice.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libdevice.so : libdvs.so: failed to map segment from shared object
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libencryption-esxcli.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libencryption-esxcli.so : libdvs.so: failed to map segment from shared object
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libfcoe.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libfcoe.so : libconfigstoreupgrade.so: failed to map segment from shared object
    Error loading plugin /usr/lib/vmware/esxcli/lib64/libgraphics.so skipping.  Error was: Unable to load plugin from /usr/lib/vmware/esxcli/lib64/libgraphics.so : libconfigstoreupgrade.so: failed to map segment from shared object
    ...
    Error when running esxcli, return status was: 1
    Errors: 
    Unable to find a command or namespace with the name 'storage' in namespace 
    Unable to find command


    [Example] From iDRAC connection and opening ESXi console : The command is excuted without problem.

    [root@localhost:~]  df -h
    Filesystem Size Used Available Use% Mounted on
    VMFSOS #.#G #.#G 1.6G 74% /vmfs/volumes/OSDATA
    vfat ###M ###M 151.1M 70% /vmfs/volumes/BOOTBANK1
    vfat ###M ###M 159.0M 68% /vmfs/volumes/BOOTBANK2
    ...

 

  • You can not find any filesystem full issue when you check 'df' command.

  • Also there is no inode exhausted if you check it with 'stat -f /' command on the ESXi host.

    [root@localhost:~] stat -f /
      File: "/"
        ID: ######## Namelen: ###  Type: visorfs
    Block size: #####
    Blocks: Total: #######  Free: #######  Available: #######
    Inodes: Total: 655360     Free: 641085  <--- The column of "Free" has enough number.


  • There are so many ssh session OOM log like below in /var/run/log/vmkernel.log file.
    [DATE/TIME] In(###) vmkernel: cpu##:########)Admission failure in path: host/vim/vimuser/terminal/ssh:python.########:uw.########
[DATE/TIME] In(###) vmkernel: cpu##:########)UserWorld 'python' ######## with cmdline 'unknown', parent ########
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'python' ######## with cmdline '/bin/python /bin/passwd', parent ########
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'sh' ######## with cmdline '-sh', parent ########
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'sshd-session' ######## with cmdline '/bin/sshd-session -i -R', parent ########
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'sshd-session' ######## with cmdline '/bin/sshd-session -i -R', parent #######
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'busybox' ####### with cmdline '/usr/lib/vmware/busybox/bin/busybox inetd /var/run/inetd.conf', parent #######
[DATE/TIME] In(###) vmkernel: cpu##:########)started from 'init' ####### with cmdline '/bin/init', parent #
[DATE/TIME] In(###) vmkernel: cpu##:########)uw.######## (##########) requires ### KB, asked ### KB from ssh (###) which has ####### KB occupied and ### KB available.
[DATE/TIME] In(###) vmkernel: cpu##:########)Admission failure in path: host/vim/vimuser/terminal/ssh:python.########:uw.########

     

  • There are so many connections from VxRail manager IP side when you check this command

    esxcli network ip connection list | grep "VxRail_Manager_IP" | wc -l

Environment

ESXi 8.0U3

VxRail environment.

Cause

This issue is not vSphere side issue but VxRail issue.

This issue is caused by a memory leak from the VxRail manager node 

Resolution

Restarting VxRail manager is solving issue in the short term.

Please contact Dell vendor for resolving the issue permanently.

Additional Information

Please refer to other issues caused by the same cause.

SSH crashes on ESXi 8.0.3 hosts in VxRail environment
https://knowledge.broadcom.com/external/article/383926/ssh-is-crashing-on-all-esxi-803-hosts-in.html

ESXi hosts lose Management IP address in iDrac and esxcli commands will fail with "MemoryError" in VxRail environments
https://knowledge.broadcom.com/external/article/405996/esxi-hosts-lose-management-ip-address-i.html

Powered by Wolken Software