Syslog over SSL forwarding fails in Usage Meter 9.1 due to a missing server.pem certificate

Products

• VMware Usage Meter

Issue/Introduction

Syslog over SSL forwarding is enabled in the Usage Meter VAMI.
The [SEND TEST MESSAGE] operation  in VAMI succeeds, but logs are not delivered to the syslog server.

The rsyslogd log on the destination syslog server outputs an error indicating that the following certificate file does not exist:
"/etc/vmware/cap/cap_am/server.pem"

Environment

Usage Meter 9.1

Cause

The specified certificate path was missing starting in Usage Meter 9.0.
However, Usage Meter 9.1 enforces strict certificate validation.
This strict enforcement generates an error due to the missing path, which ultimately causes the syslog forwarding to fail.

Resolution

This issue can be resolved by specifying a valid certificate path.
The steps are as follows:

1. Take an offline snapshot of the Usage Meter appliance.
Take the snapshot using the vSphere Client or Host Client.
(Use this snapshot to revert if the operation fails. Additionally,
delete this snapshot once the procedure is complete and syslog forwarding is functioning normally.)

2. Connect to the Usage Meter appliance via SSH, log in as the usagemeter user, and then elevate to the root user.
 
3. Ensure that the directory exists.
# mkdir -p /etc/vmware/cap/cap_am

4. Generate a private key and a self-signed certificate for the Usage Meter appliance.
(This fulfills the requirements for the CertFile and KeyFile configuration directives).

# openssl req -x509 -nodes -newkey rsa:2048 \
  -keyout /tmp/um.key -out /tmp/um.crt -days 3650 \
  -subj "/C=US/ST=California/L=Palo Alto/O=VMware, Inc./OU=Usage Meter/CN=<UM_FQDN>"

5. Retrieve the current public certificate from the syslog server
(to add as a trust anchor for the CAFile directive).

# echo | openssl s_client -connect <syslog-ip>:<syslog-port> 2>/dev/null | openssl x509 > /tmp/syslog.crt

6. Combine the private key, the local certificate, and the syslog certificate into the single file referenced by the configuration directives.
(Ensure the private key is placed first, followed by the certificates.)

# cat /tmp/um.key /tmp/um.crt /tmp/syslog.crt > /etc/vmware/cap/cap_am/server.pem

7. Restrict the file permissions, as this file now contains a sensitive private key.

# chmod 600 /etc/vmware/cap/cap_am/server.pem
# chown root:root /etc/vmware/cap/cap_am/server.pem

Verify the user account running the rsyslogd service (in the event it is not running as root):

# systemctl show rsyslog -p User -p Group

8. Restart the rsyslog service to load the updated configuration.

# systemctl restart rsyslog

9. Confirming that the certificate errors are no longer present in the logs.

# journalctl -u rsyslog --since "-2min" | grep -i "error\|suspend\|resum"

10. Verify that log forwarding to the destination syslog server is successful.