Unable to send emails via relay SMTP after replacing vCenter certificates

Products

• VMware vCenter Server

Issue/Introduction

Logs show errors similar to the excerpts below:

/var/log/vmware/messages
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sudo:     vpxd : PWD=/storage/log/vmware/vpxd ; USER=root ; COMMAND=/bin/sudo_command_wrapper.sh /usr/sbin/sendmail -tf <email_adddress> 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1014) 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: <service_account>: from=<email_adddress>, size=931, class=0, nrcpts=1, msgid=<202510081325.<service_account>@<vc_fqdn>.<domain>>, 
relay=root@localhost 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: STARTTLS=client, relay=<relay>., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: <service_account>: to=<email_adddress>, ctladdr=<email_adddress> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30931, 
relay=<relay>. [<ip_address>], dsn=5.7.1, reply=554 5.7.1 <<email_adddress>>: 
Relay access denied, stat=Service unavailable 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: <service_account>: 598DPpuV2565374: DSN: Service unavailable 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: ###############: to=<email_adddress>, delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=31955, 
relay=<relay>. [<ip_address>], dsn=5.7.1, reply=554 5.7.1 <<email_adddress>>: Relay access denied, stat=Service unavailable 
yyyy-mm-ddThh:mm:ss.zzz+00:00 <vc_fqdn> sendmail[2565374]: ###############: 598DPpuW2565374: return to sender: Service unavailable 

Environment

VMware vCenter Server 8.x

Cause

This is a know issue affecting vCenter 8.x causing STARTTLS to not work correctly.

Issue occurs because the system certificates sendmail is configured to use are not properly updated in /etc/applmgmt/sendmail/

Resolution

The issue is resolved in vCenter 9.x.

 

Workaround:

1. Take a snapshot of the vCenter Virtual machine. If the environment is in ELM topology check KB313886
   
   
2. SSH to vCenter as root user.
   
   
3. Create necessary folders and certificates by executing the following commands:
   
   mkdir -p /etc/applmgmt/sendmail
cp -L /etc/vmware/vmware-vmafd/ca.crt /etc/applmgmt/sendmail/ca.pem
cp -L /etc/vmware/vmware-vmafd/machine-ssl.crt /etc/applmgmt/sendmail/server.cert.pem
cp -L /etc/vmware/vmware-vmafd/machine-ssl.key /etc/applmgmt/sendmail/server.key.pem
chown -R root:root /etc/applmgmt/sendmail
chmod 600 /etc/applmgmt/sendmail/server.cert.pem
chmod 600 /etc/applmgmt/sendmail/server.key.pem
chmod 644 /etc/applmgmt/sendmail/ca.pem
   
   
4. Update the sendmail.mc file using the template below:
   
   

   ######################################################################
# /etc/mail/sendmail.cf
#
#
######################################################################
divert(-1)
include(`/etc/mail/m4/cf.m4')
divert(0)dnl
VERSIONID(`@(#)Setup for Generic Linux')dnl
OSTYPE(`linux')dnl
define(`confPRIVACY_FLAGS', `noexpn,novrfy')dnl
define(`confLOG_LEVEL', `98')dnl

   define(`SMART_HOST', `[<server>]')dnl
define(`RELAY_MAILER_ARGS', `TCP $h <port>')dnl
define(`ESMTP_MAILER_ARGS', `TCP $h <port>')dnl

   # certs for STARTTLS
define(`confCACERT_PATH', `/etc/applmgmt/sendmail')dnl
define(`confCACERT', `/etc/applmgmt/sendmail/ca.pem')dnl
define(`confSERVER_CERT', `/etc/applmgmt/sendmail/server.cert.pem')dnl
define(`confSERVER_KEY', `/etc/applmgmt/sendmail/server.key.pem')dnl
define(`confCLIENT_CERT', `/etc/applmgmt/sendmail/server.cert.pem')dnl
define(`confCLIENT_KEY', `/etc/applmgmt/sendmail/server.key.pem')dnl

   LOCAL_CONFIG
O CipherSuites=TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256
O CipherList=ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-SHA:AES128-SHA
O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1 +SSL_OP_NO_TLSv1_1
O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1 +SSL_OP_NO_TLSv1_1

   FEATURE(`authinfo', `cdb /etc/mail/auth/auth-info')dnl
define(`confAUTH_MECHANISMS', `login,plain')dnl
TRUST_AUTH_MECH(`login,plain')dnl

   FEATURE(`no_default_msa')dnl
FEATURE(`accept_unresolvable_domains')dnl
FEATURE(`always_add_domain')dnl
FEATURE(`generics_entire_domain')dnl
GENERICS_DOMAIN_FILE(`-o /etc/mail/local-host-names')dnl
DOMAIN(`generic')dnl
MAILER(`local')dnl
MAILER(`smtp')dnl
MAILER(`procmail')dnl
MAILER(`uucp')dnl


5. Make sure to update in the above the fields
   <server>
   with actual smtp server
   <port>
   with actual port number
   
   The auth mechanism
   Currently the default is login,plain
   
   sendmail supports the following methods: plain,login,gssapi,digest-md5,cram-md5
   
   If you need to change the method make the necessary changes in these 2 lines in the template:
   
   define(`confAUTH_MECHANISMS', `login,plain')dnl

   and
   
   TRUST_AUTH_MECH(`login,plain')dnl



6. Save this file as /etc/mail/sendmail.mc



    

7. Verify the details in authinfo. Make sure that the auth info file `/etc/mail/auth/auth-info` contains the appropriate login credentials. Then generate the auth-info.cdb file using below command:
   
   /usr/sbin/makemap cdb /etc/mail/auth/auth-info.cdb < /etc/mail/auth/auth-info

    

8. Rebuild the sendmail.cf file:
   
   /usr/bin/m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

    

9. Restart the sendmail
   
   /usr/bin/systemctl restart sendmail

    

10. Trigger the mail:
    
    TO=<TO_ADDRESS> FROM=<FROM_ADDRESS>; /usr/sbin/sendmail -i -v -Am -d -tf $FROM > /var/log/vmware/sendmail_$(date '+%FT%T%:z').log -- $TO <<END
FROM: $FROM
Subject: Sendmail Delivery Test Using Relay Server
To: $TO
Delivery test [$(date '+%FT%T%:z')].
END

    Note: Change the fields <TO_ADDRESS> and <FROM_ADDRESS> to appropriate strings.