vpxd-svcs
" service./var/log/vmware/applmgmt/Patchrunner.log
will show entries similar to :INFO service_manager Service vmware-vpxd-svcs reported status stopped. Expected status started
ERROR service_manager Service cannot be started. Error: Error executing start on service vpxd-svcs. Details {
File "/storage/seat/software-updatedrthhh7b/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
super(VMwareServiceController, self).start(serviceName)
File "/storage/seat/software-updatedrthhh7b/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd-svcs.
/var/log/vmware/vmon/vmon.log
will show vpxd-svcs pre-start failure with "Connection refused
" error message:In(05) host-36832 Received start request for vpxd-svcs
In(05) host-36832 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
Wa(03) host-36832 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
Wa(03)+ host-36832 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
Wa(03)+ host-36832 endpoint_registration_runner(logging_file)
Wa(03)+ host-36832 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
Wa(03)+ host-36832 UpdateTaggingServiceGrpcEndpoint(logging_file).run()
Wa(03)+ host-36832 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
Wa(03)+ host-36832 self.update_endpoints()
Wa(03)+ host-36832 File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 83, in update_endpoints
Wa(03)+ host-36832 ls_obj = LookupServiceClient(ls_url, retry_count=5)
Wa(03)+ host-36832 File "/usr/lib/python3.7/socket.py", line 727, in create_connection
Wa(03)+ host-36832 raise err
Wa(03)+ host-36832 File "/usr/lib/python3.7/socket.py", line 716, in create_connection
Wa(03)+ host-36832 sock.connect(sa)
Wa(03)+ host-36832 ConnectionRefusedError: [Errno 111] Connection refused
Wa(03)+ host-36832
Er(02) host-36832 <vpxd-svcs> Service pre-start command failed with exit code 1.
/var/log/vmware/envoy/envoy.log
will show TLS cipher initialization failures as below :The following ciphers were rejected when tried individually: aNULL, kECDH, /external-vecs/http1/<VC FQDN>/443/: Failed to initialize cipher suites !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM.
This issue is caused due to custom configuration of TLS cipher suites in the file /etc/vmware-rhttpproxy/config.xml
. Envoy service on vCenter Server 7.0 U3q is built on BoringSSL instead of OpenSSL as the previous releases, OpenSSL prior to version 1.1.0 silently ignores the unsupported ciphers. However, the version of BoringSSL used in 7.0 U3q does not accept unsupported ciphers and fails to initialize.
To resolve the issue, reset the TLS Ciphers to the defaults and retry the vCenter Server update.
If required, configure customized ciphers post vCenter Server upgrade (refer to Additional Information section of this KB for more details).
Steps to Reset TLS Ciphers to the default values.
Scripted method to reset the ciphers.
reset_ciphers_kb369485.py
" attached to this KB/root
or /tmp
location using WinSCP or Copy/Paste using vi
editoryes
' to continue with the script
Manual method to reset the ciphers.
rhttpproxy
serviceservice-control --stop rhttpproxy
/etc/vmware-rhttpproxy/config.xml
and remove the line <cipherlist>list of ciphers</cipherlist>
vi /etc/vmware-rhttpproxy/config.xml
<cipherlist>
till </cipherList> :
<config>
<vmacore>
.
.
<threadPool>
.
</threadPool>
<ssl>
<cipherList>!aNULL:ECDH+AES:!ECDHE-RSA-AES128-SHA256:!ECDHE-RSA-AES128-SHA:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-CHAC</cipherList>
<doVersionCheck> false </doVersionCheck>
<!-- allowed SSL/TLS protocol versions -->
<!-- <protocols>tls1.0,tls1.1,tls1.2</protocols> -->
<fips>true</fips>
<libraryPath></libraryPath>
</ssl>
wq!
in vi
editor) to configuration file config.xml:
Final changes should look like this :
<config>
<vmacore>
.
.
<threadPool>
.
</threadPool>
<ssl>
<doVersionCheck> false </doVersionCheck>
<!-- allowed SSL/TLS protocol versions -->
<!-- <protocols>tls1.0,tls1.1,tls1.2</protocols> -->
<fips>true</fips>
<libraryPath></libraryPath>
</ssl>
cp /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json ~/ProxyConfiguration.json.bak
cat /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json | jq . | sed 's/"cipher_suites": *"[^"]*"/"cipher_suites": "ECDHE+AESGCM"/g' > ~/ProxyConfiguration.json.new
mv ~/ProxyConfiguration.json.new /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json
rhttpproxy
serviceservice-control --start rhttpproxy
Configure Custom Ciphers on Port 443
There are chances of security scanning softwares reporting warnings for Weak ciphers after resetting the values to default. You may disable these ciphers by resetting the configuration back to Custom values which are supported by BoringSSL on VCSA 7.0 U3q or higher versions.
Add the following cipher list to the configuration file /etc/vmware-rhttpproxy/config.xml
to restrict the ciphers :
<cipherList>ECDHE+AESGCM</cipherList>
Note: Only following ciphers will be enabled on Port 443 after setting above cipherlist:
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-GCM-SHA256
You may refer to KB Disabling static ciphers for TLS in the vCenter Server appliance for more information to update the configuration file.