Adding an ESXi 5.5 host with a SSH banner to a cluster fails with: A general system error occurred

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

When attempting to add an ESXi host to the cluster you get errors similar to:

A general system error occurred: Timed waiting for vpxa to start
A general system error occurred: Internal error: vmodl.fault.HostCommunication

The ESXi host can be added to the data center outside of the cluster
The vpxa.log file located at /var/log/ displays entries similar to:

<YYYY-MM-DD><time>[FFA54B70 warning 'Default'] Closing Response processing in unexpected state: 3 
<YYYY-MM-DD><time>[FFA54B70 verbose 'VpxaHalCnxHostagent' opID=WFU-f4acae3c] [WaitForUpdatesDone] Received callback 
<YYYY-MM-DD><time>[FFA54B70 error 'VpxaHalCnxHostagent' opID=WFU-f4acae3c] [WaitForUpdatesDone] Got error in WaitForUpdates() call to hostd: vmodl.fault.SystemError 
<YYYY-MM-DD><time>[FFA54B70 error 'VpxaHalCnxHostagent' opID=WFU-f4acae3c] [WaitForUpdatesDone] Fatal error while listening-for/processing updates from hostd. Backtrace: 
--> backtrace[00] rip 16426113 Vmacore::System::Stacktrace::CaptureWork(unsigned int) 
--> backtrace[01] rip 1627ea1b Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&) 
--> backtrace[02] rip 1621fe15 Vmacore::Throwable::Throwable(std::string const&) 
--> backtrace[03] rip 1684e0df Vmomi::SoapScheduledItemImpl::GetResultFromResponse(Vmacore::Http::ClientResponse*, Vmomi::SoapStubAdapterImpl*, Vmacore::Ref<Vmomi::Any>&) const 
--> backtrace[04] rip 1684f3f7 Vmomi::SoapScheduledItemImpl::RequestCompletedImpl(Vmacore::Exception*, Vmacore::Http::ClientResponse*, Vmacore::Http::PendingRequestItem*) 
--> backtrace[05] rip 16857d95 Vmomi::SoapScheduledItemImpl::RequestCompleteParams::Invoke() 
--> backtrace[06] rip 16853ad3 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, Vmomi::SoapScheduledItemImpl::RequestCompleteParams>, boost::_bi::list 
1<boost::_bi::value<Vmacore::Ref<Vmomi::SoapScheduledItemImpl::RequestCompleteParams> > > >, void>::invoke(boost::detail::function::function_buffer&) 
--> backtrace[07] rip 163876ca Vmacore::System::ThreadPoolAsio::DoWork(Vmacore::Functor<void, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil> const&, bool) 
--> backtrace[08] rip 16387f08 Vmacore::System::ThreadPoolAsio::DoOrEnqueueWork(Vmacore::System::ThreadPoolAsio::WorkTypeInfo&, Vmacore::Functor<void, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore: <br>:Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil> const&, int) 
--> backtrace[09] rip 16388170 Vmacore::System::ThreadPoolAsio::HandleWork(Vmacore::System::ThreadPoolAsio::WorkType, Vmacore::Functor<void, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil, Vmacore::Nil> const&) 
--> backtrace[10] rip 16390421 Vmacore::System::ThreadPoolAsioWorkBinder::InvokeWork_0(Vmacore::System::ThreadPoolAsio::WorkType, boost::function<void ()> const&) 
--> backtrace[11] rip 1638be7a boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, ()&gt; (*)(vmacore::system::threadpoolasio::worktype, boost::_bi::list2&lt;boost::_bi::value&lt;vmacore::system::threadpoolasio::worktype&gt;, boost::_bi::value<boost::function&lt;void boost::function&lt;void const&), void> > >, void>::invoke(boost::detail::function::function_buffer&) 
--> backtrace[12] rip 16391efc boost::asio::detail::completion_handler<boost::function&lt;void ()&gt;>::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code, unsigned int) 
--> backtrace[13] rip 1638f20e boost::asio::detail::task_io_service::do_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service::idle_thread_info*) 
--> backtrace[14] rip 16386e0c Vmacore::System::ThreadPoolAsio::RunWorkerThread() 
--> backtrace[15] rip 1638bed3 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, Vmacore::System::ThreadPoolAsio>, boost::_bi::list1<boost::_bi::value< <br>Vmacore::Ref<Vmacore::System::ThreadPoolAsio> > > >, void>::invoke(boost::detail::function::function_buffer&) 
--> backtrace[16] rip 1642fc94 Vmacore::System::ThreadPosixPrivate::ThreadBegin(void*) 
--> backtrace[17] rip 15eb4d6a /lib/libpthread.so.0(+0x6d6a) [0x15eb4d6a] 
--> 
</boost::function&lt;void></boost::_bi::bind_t<void,></time></time></time></time><YYYY-MM-DD><time></time>[FFA54B70 error 'vpxavpxaInvtHostCnx' opID=WFU-f4acae3c] [VpxaInvtHost] Can't connect to hostd/serverd. Shutting down... 
<YYYY-MM-DD><time>[FFA54B70 info 'Default' opID=WFU-f4acae3c] [Vpxa] Shutting down now  </time>OR:  <YYYY-MM-DD><time> [28869B70 error 'vmomi.soapStub[2]']  </time>

--> Error returned by expat parser: not well-formed (invalid token) 
--> 
--> while parsing serialized value of type string 
--> error parsing Any with xsiType string 
--> at line 7, column 101560 
--> 
--> while parsing property "value" of static type anyType 
--> 
--> while parsing serialized DataObject of type vim.option.OptionValue 
--> at line 7, column 101497 
--> 
--> error parsing Any with xsiType ArrayOfOptionValue 
--> at line 7, column 92206 
--> 
--> while parsing property "val" of static type anyType 
--> 
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.Change 
--> at line 7, column 92160 
--> 
--> while parsing property "changeSet" of static type ArrayOfPropertyChange 
--> 
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectUpdate 
--> at line 7, column 92085 
--> 
--> while parsing property "objectSet" of static type ArrayOfObjectUpdate 
--> 
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterUpdate 
--> at line 7, column 75116 
--> 
--> while parsing property "filterSet" of static type ArrayOfPropertyFilterUpdate 
--> 
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.UpdateSet 
--> at line 7, column 42 
--> 
--> while parsing return value of type vmodl.query.PropertyCollector.UpdateSet, version vim.version.version9 
--> at line 7, column 0 
--> 
--> while parsing SOAP body 
--> at line 6, column 0 
--> 
--> while parsing SOAP envelope 
--> at line 2, column 0 
--> 
--> while parsing HTTP response for method waitForUpdates 
--> on object of type vmodl.query.PropertyCollector 
--> at line 1, column 0 
<YYYY-MM-DD><time>[28869B70 warning 'Default'] Closing Response processing in unexpected state: 3 </time>

Environment

VMware vSphere ESXi 5.5

Cause

This issue is caused when the ESXi hosts in the cluster are using a logon banner and there is a mismatch in the logon banner file between the ESXi hosts.

Resolution

To resolve this issue the SSH banner files must be the same size on all ESXi hosts in the cluster.

To verify the size of the SSH banner file:

Connect to the ESXi host through an SSH session. For more information, see Using ESXi Shell in ESXi 5.x (2004746).
Type ls -l /etc/issue to check the size of the SSH banner file.

Example Output:

-rw-rw-rwT 1 root root 10 Feb 22 01:28 /etc/issue
Compare this size to the other ESXi hosts in the cluster.
If it is a different size create a backup of the file with:

cp /etc/issue /etc/issue.bkp
Copy the issue file from a working host replacing the existing issue file.
Restart the SSH daemon using this command:

/etc/init.d/SSH restart

For more information on the SSH Banner, see Enabling the pre-login SSH banner in ESXi 5.x (2068379)

Additional Information

Using ESXi Shell in ESXi 5.x and 6.x

Enabling the pre-login SSH banner in ESXi 5.x