VMware Aria Operations Displays "Failed to start the Apache HTTP Server." On the Console Screen During Boot

Products

• VMware Aria Suite

Issue/Introduction

During the upgrade of VMware Aria Operations to version 8.14 or higher, the appliance VM is rebooted but neither the Admin UI or Product UI become available again after the reboot. During the boot process of the appliance, a message is seen stating "[FAILED] Failed to start the Apache HTTP Server."

Environment

VMware Aria Operations 8.10 upgrading to 8.14

Cause

The "[FAILED] Failed to start the Apache HTTP Server." error during appliance boot can be caused by the presence of the SSLFIPS on directive within the httpd.conf configuration file for the Apache HTTP Server in VMware Aria Operations 8.14 and higher. To verify if the issue is present, use the following steps.

1. Log in to the primary node as root via SSH or Console
2. Run the command:

grep "SSLFIPS on" /etc/httpd/conf/httpd.conf

If the command returns SSLFIPS on, continue to the steps in the Resolution section below.

If the command returns no results, repeat steps 1 and 2 on all analytic nodes (primary replica or data) in the VMware Aria Operations cluster.

If the command returns no results on all nodes, the issue could be a bug which is resolved in 8.14.1 where the upgrade hangs and produces the same error. See the link below;

• Aria Operations 8.14.1 fix list.

Resolution

To successfully upgrade the VMware Aria Operations cluster to version 8.14 or higher, the SSLFIPS on directive must be removed from the httpd.conf configuration file for the Apache HTTP Server prior to the upgrade attempt using the steps below.

1. Revert all nodes in the VMware Aria Operations cluster to the snapshot that was taken prior to the upgrade attempt.

Note: If the pre-upgrade snapshots are no longer available, restore all nodes in the VMware Aria Operations cluster including cloud proxies from an existing backup that was taken before the upgrade was attempted.

2. Power on all the VMware Aria Operations appliances using the sequence from the Bringing Nodes Online section for the cluster type as detailed in Shutdown and Startup sequence for a vRealize Operations cluster (341964)
3. Log in to the primary node as root via SSH or Console
4. Remove the SSLFIPS on directive from the httpd.conf configuration file using the command:

sed -i 's\SSLFIPS on\\' /etc/httpd/conf/httpd.conf

5. Restart the Apache HTTP Server with the command:

systemctl restart httpd

6. Repeat steps 3 through 5 on all analytic nodes (primary replica and data) in the VMware Aria Operations cluster.
7. Use the steps from Install a Software Update to complete the upgrade of the VMware Aria Operations cluster to version 8.14 or higher.

Note: To enable FIPS 140-2 mode in VMware Aria Operations, use the steps from Activating FIPS 140-2

If issue doesn't get resolved by above mentioned steps, then follow below:

1. Go to /usr/lib/vmware-vcopssuite/utilities/conf directory in primary node.
2. Open the file using cat vcops-apache.conf command and check if TLS version entry is correct or not.

 Sample content below:    

   SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM- SHA256:ECDHE-RSA-AES256-GCM-SHA384:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
   # Enable all available TLS flavours, disable SSLv2, SSLv3, and TLSv1
   # Until openssl and httpd are upgraded, TLSv1.1 and TLSv1.2 cannot be added here.  This subtractive method will allow the newer TLS protocols when available.

      SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
   SSLHonorCipherOrder on
   TraceEnable off

   ExpiresActive On

Refer: Verify the Correct Use of Protocols in Apache HTTPD

     3. Restart Apache service using systemctl restart httpd command.

     4. Repeat above steps on all nodes.

 

Note: If all the above steps don't resolve the issue, please upgrade to 8.14.1, following the usual upgrade process.