Failed to modify any of the ESXi host SSH configuration using either esxcli or localcli commands
search cancel

Failed to modify any of the ESXi host SSH configuration using either esxcli or localcli commands

book

Article ID: 413803

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 8.0

Issue/Introduction

  • Users encounter automation failures on one or more ESXi hosts when they run a custom Python script that executes an esxcli command.
  • The script runs under a dedicated custom user account with the Administrator role, which holds privileges equivalent to the default root account in VMware ESXi.
  • The execution consistently displays the following error 
    Failed to validate config file /etc/ssh/sshd_config.tmp.######
  • When users perform the same operation manually - either through the ESXi Host DCUI or over SSH using the root account and run the corresponding esxcli or localcli commands, the outcome remains the same, resulting in the identical error.
  • Sample reference commands used to modify the ESXi host SSH configuration include -
    • esxcli system ssh server config set -k loglevel -v debug OR localcli system ssh server config set -k loglevel -v debug
    • esxcli system ssh server config set -k permitrootlogin -v no OR localcli system ssh server config set -k permitrootlogin -v no
  • This issue specifically affects modifications to the ESXi host SSH configuration made through esxcli commands, regardless of whether the operation is automated or performed manually.
  • The entries in the /var/run/log/localcli.log are as follows 
    In(14) localcli[210####]: Util: ForkExec(/usr/lib/vmware/openssh/bin/sshd   -t -f /etc/ssh/sshd_config.tmp.EYc###) 2101###
Er(11) localcli[210####]: SshServerConfigImpl: Validation failed : /etc/ssh/sshd_config.tmp.EYc###
Er(11) localcli[210####]: SshServerConfigImpl: Failed to save config: Failed to validate config file /etc/ssh/sshd_config.tmp.EYc###
  • The entries in the /var/run/log/hostd.log are as follows  
    Hostd[21043##]: [Originator@6876 sub=Libs opID=esxcli-b9-#### sid=5261#### user=root] Util: ForkExec(/usr/lib/vmware/openssh/bin/sshd  ++group=hostd-tmp,mem=120 -t -f /etc/ssh/sshd_config.tmp.MCZ### ) 2104###
Hostd[21043##]: [Originator@6876 sub=Libs opID=esxcli-b9-#### sid=5261#### user=root] SshServerConfigImpl: Validation failed : /etc/ssh/sshd_config.tmp.MCZ###
Hostd[21043##]: [Originator@6876 sub=Libs opID=esxcli-b9-#### sid=5261#### user=root] SshServerConfigImpl: Failed to save config: Failed to validate config file /etc/ssh/sshd_config.tmp.MCZ###
Hostd[21043##]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-b9-#### sid=5261#### user=root] Dispatch system.ssh.server.config.set done
Hostd[21043##]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-b9-#### sid=5261#### user=root] Dispatch system.ssh.server.config.set failed
Hostd[21043##]: [Originator@6876 sub=AdapterServer opID=esxcli-b9-#### sid=5261#### user=root] AdapterServer caught exception; <<5261####-####-####-e04e-7b50a1######, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 15115'>>, h
Hostd[21042##]: --> )
Hostd[21042##]: --> [context]zKq7#####jYA[/context]
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Activation finished; <<5261####-####-####-e04e-7b50a1######, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 15115'>>, ha-cli-handler-
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Arg keyword:
Hostd[21042##]: --> "loglevel"
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Arg reset:
Hostd[21042##]: --> (null)
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Arg value:
Hostd[21042##]: --> (string) [
Hostd[21042##]: -->    "debug"
Hostd[21042##]: --> ]
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Throw vim.EsxCLI.CLIFault
Hostd[21043##]: [Originator@6876 sub=Solo.Vmomi opID=esxcli-b9-#### sid=5261#### user=root] Result:
Hostd[21042##]: --> (vim.EsxCLI.CLIFault) {
Hostd[21042##]: -->    errMsg = (string) [
Hostd[21042##]: -->       "Failed to validate config file /etc/ssh/sshd_config.tmp.MCZ###"
Hostd[21042##]: -->    ],
Hostd[21042##]: -->    msg = "",
Hostd[21042##]: --> }


Environment

  • VMware vSphere ESXi 8.0.x

Cause

  • The unaffected ESXi host /etc/ssh/sshd_config file content - 

# Do not edit this file, config store overwrites it.
# This file contains the Server Config for SSH.
# Running from inetd.
...
...
syslogfacility auth
tcpkeepalive yes
usepam yes
...
...
authorizedkeysfile /etc/ssh/keys-%u/authorized_keys
PasswordAuthentication no

  • The affected ESXi host /etc/ssh/sshd_config file content - 

# Do not edit this file, config store overwrites it.
# This file contains the Server Config for SSH.
# Running from inetd.
...
...
syslogfacility auth
tcpkeepalive yes
usepam yes
...
...
authorizedkeysfile /etc/ssh/keys-%u/authorized_keys
PasswordAuthentication no
UsePAM no

  • The key difference between the unaffected ESXi host and the affected ESXi host /etc/ssh/sshd_config files is the presence of both - lowercase parameter usepam yes in both hosts, and an additional uppercase parameter UsePAM no only in the affected host. The issue arises from conflicting SSH configuration parameters in the /etc/ssh/sshd_config file, where newer ESXi versions (vSphere 8.x and later) require the lowercase 'usepam' parameter, while older versions (vSphere 7.0 and earlier) use the uppercase 'UsePAM' parameter by default. Having both entries simultaneously causes validation failures during SSH configuration updates.

Resolution

To address and prevent recurrence of this issue, the following corrective actions are recommended - 

  1. SSH to the affected ESXi host using root user credentials
  2. Run the following command to view the current SSH configuration file content -
    cat /etc/ssh/sshd_config

    Sample output - 
    # Do not edit this file, config store overwrites it.
# This file contains the Server Config for SSH.
# Running from inetd.
...
...
syslogfacility auth
tcpkeepalive yes
usepam yes
...
...
authorizedkeysfile /etc/ssh/keys-%u/authorized_keys
PasswordAuthentication no
UsePAM no


    Please Note -  The parameter usepam is duplicated here, once in lowercase and once in uppercase.

  3. Create a backup of the sshd_config file by executing the following command -
    cp /etc/ssh/sshd_config /tmp/sshd_config_bkp
  4. Modify the file and  remove the uppercase value for usepam, using the vi text editor as shown below - 
    vi /etc/ssh/sshd_config

    Sample sshd_config file after update

    # Do not edit this file, config store overwrites it.
# This file contains the Server Config for SSH.
# Running from inetd.
...
...
syslogfacility auth
tcpkeepalive yes
usepam yes
...
...
authorizedkeysfile /etc/ssh/keys-%u/authorized_keys
PasswordAuthentication no


  5. Save the changes and quit by pressing Esc, then typing ':wq!' and hitting Enter.
  6. Restart the sshd service on the ESXi host using the below command -
    /etc/init.d/SSH restart
  7. Retry the SSHD configuration commands used to modify the ESXi host's SSH configuration, such as -
    esxcli system ssh server config set -k loglevel -v debug OR esxcli system ssh server config set -k permitrootlogin -v no
Powered by Wolken Software