"An error occurred while starting service 'vpxd-svcs'", vCenter Server patching to 7.0 U3q fails due to unsupported TLS Ciphers in Envoy Configuration
search cancel

"An error occurred while starting service 'vpxd-svcs'", vCenter Server patching to 7.0 U3q fails due to unsupported TLS Ciphers in Envoy Configuration

book

Article ID: 369485

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

  • Patching vCenter Server to 7.0 U3q (7.0.3.01900) fails while starting "vpxd-svcs" service.

  • VAMI will show the error message "Installation Failed - Exception occurred in postInstallHook".



  • Log file /var/log/vmware/applmgmt/Patchrunner.log will show entries similar to :

    INFO service_manager Service vmware-vpxd-svcs reported status stopped. Expected status started
    ERROR service_manager Service cannot be started. Error: Error executing start on service vpxd-svcs. Details {
      File "/storage/seat/software-updatedrthhh7b/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
        super(VMwareServiceController, self).start(serviceName)
      File "/storage/seat/software-updatedrthhh7b/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
        raise IllegalServiceOperation(errorText)
    service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd-svcs.

  • vMon log file /var/log/vmware/vmon/vmon.log will show vpxd-svcs pre-start failure with "Connection refused" error message:

    In(05) host-36832 Received start request for vpxd-svcs
    In(05) host-36832 <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
    Wa(03) host-36832 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
    Wa(03)+ host-36832   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
    Wa(03)+ host-36832     endpoint_registration_runner(logging_file)
    Wa(03)+ host-36832   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
    Wa(03)+ host-36832     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
    Wa(03)+ host-36832   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
    Wa(03)+ host-36832     self.update_endpoints()
    Wa(03)+ host-36832   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 83, in update_endpoints
    Wa(03)+ host-36832     ls_obj = LookupServiceClient(ls_url, retry_count=5)
    Wa(03)+ host-36832   File "/usr/lib/python3.7/socket.py", line 727, in create_connection
    Wa(03)+ host-36832     raise err
    Wa(03)+ host-36832   File "/usr/lib/python3.7/socket.py", line 716, in create_connection
    Wa(03)+ host-36832     sock.connect(sa)
    Wa(03)+ host-36832 ConnectionRefusedError: [Errno 111] Connection refused
    Wa(03)+ host-36832
    Er(02) host-36832 <vpxd-svcs> Service pre-start command failed with exit code 1.

  • Envoy service log /var/log/vmware/envoy/envoy.log will show TLS cipher initialization failures as below :

    The following ciphers were rejected when tried individually: aNULL, kECDH, /external-vecs/http1/<VC FQDN>/443/: Failed to initialize cipher suites !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM.

Cause

This issue is caused due to custom configuration of TLS cipher suites in the file /etc/vmware-rhttpproxy/config.xml. Envoy service on vCenter Server 7.0 U3q is built on BoringSSL instead of OpenSSL as the previous releases, OpenSSL prior to version 1.1.0 silently ignores the unsupported ciphers. However, the version of BoringSSL used in 7.0 U3q does not accept unsupported ciphers and fails to initialize.

Resolution

To resolve the issue, reset the TLS Ciphers to the defaults and retry the vCenter Server update.

If required, configure customized ciphers post vCenter Server upgrade (refer to Additional Information section of this KB for more details).

Steps to Reset TLS Ciphers to the default values.

 

Scripted method to reset the ciphers.

  1. Download the script "reset_ciphers_kb369485.py" attached to this KB
  2. Connect to vCenter Server through SSH
  3. Copy the script to vCenter Server /root or /tmp location using WinSCP or Copy/Paste using vi editor
  4. Execute the script using Python and enter 'yes' to continue with the script

    Sample Result:

 

Manual method to reset the ciphers.

  1. Connect to vCenter Server through SSH
  2. Stop the rhttpproxy service

    service-control --stop rhttpproxy

  3. Edit the config file /etc/vmware-rhttpproxy/config.xml and remove the line <cipherlist>list of ciphers</cipherlist>

    vi /etc/vmware-rhttpproxy/config.xml

    Remove the entire line highlighted in yellow starting with <cipherlist> till </cipherList> :

    Note: The cipherlist mentioned below is a sample list, it might change based on the Custom configuration in specific environments.

    <config>
      <vmacore>
        .
        .
        <threadPool>
          .
        </threadPool>
        <ssl>
          <cipherList>!aNULL:ECDH+AES:!ECDHE-RSA-AES128-SHA256:!ECDHE-RSA-AES128-SHA:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-CHAC</cipherList>
          <doVersionCheck> false </doVersionCheck>
          <!-- allowed SSL/TLS protocol versions -->
          <!-- <protocols>tls1.0,tls1.1,tls1.2</protocols> -->
          <fips>true</fips>
          <libraryPath></libraryPath>
        </ssl>

    Save the changes (wq! in vi editor) to configuration file config.xml:

    Final changes should look like this :

    <config>
      <vmacore>
        .
        .
        <threadPool>
          .
        </threadPool>
        <ssl>
          <doVersionCheck> false </doVersionCheck>
          <!-- allowed SSL/TLS protocol versions -->
          <!-- <protocols>tls1.0,tls1.1,tls1.2</protocols> -->
          <fips>true</fips>
          <libraryPath></libraryPath>
        </ssl>

  4. Edit the file /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json

    Create backup of ProxyConfiguration.json:

    cp /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json ~/ProxyConfiguration.json.bak

    Modify value of cipher_suites field to ECDHE+AESGCM:

    cat /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json | jq . | sed 's/"cipher_suites": *"[^"]*"/"cipher_suites": "ECDHE+AESGCM"/g' > ~/ProxyConfiguration.json.new

    Replace the configuration ProxyConfiguration.json with the updated file:
     
    mv ~/ProxyConfiguration.json.new /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json

  5. Start the rhttpproxy service

    service-control --start rhttpproxy

Additional Information

Configure Custom Ciphers on Port 443

There are chances of security scanning softwares reporting warnings for Weak ciphers after resetting the values to default. You may disable these ciphers by resetting the configuration back to Custom values which are supported by BoringSSL on VCSA 7.0 U3q or higher versions.

Add the following cipher list to the configuration file /etc/vmware-rhttpproxy/config.xml to restrict the ciphers :

<cipherList>ECDHE+AESGCM</cipherList>

Note: Only following ciphers will be enabled on Port 443 after setting above cipherlist:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-GCM-SHA256

You may refer to KB Disabling static ciphers for TLS in the vCenter Server appliance for more information to update the configuration file.

Attachments

reset_ciphers_kb369485.py get_app
Powered by Wolken Software