Sending Emails via 'Notifications' Service Fails in Pivotal Cloud Foundry

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

When an invitation to a new user is sent via the app-manager/console UI, the user does not get an email notification.

Looking at the logs via cf cli, we see the following errors:

-> cf logs notifications --recent

--snip--
2015-06-24T16:04:56.86-0500 [App/0]      ERR [MySQL] 2015/06/24 21:04:56 packets.go:118: write tcp 10.17.128.51:3306: broken pipe
2015-06-24T16:04:56.88-0500 [RTR/0]      OUT notifications.xyz.com - [24/06/2015:21:04:56 +0000] "POST /emails HTTP/1.1" 200 117 "-" "Typhoeus - https://github.com/typhoeus/typhoeus" 10.17.128.250:33931 x_forwarded_for:"10.17.128.254" vcap_request_id:8b5845e9-0380-4a1f-7577-6162547ad283 response_time:0.031032076 app_id:a15abe61-9666-4b87-bc15-37ca534098b3
2015-06-24T16:04:56.96-0500 [App/0]      OUT [WEB] 2015/06/24 21:04:56 Attempting to deliver message to [email protected]
2015-06-24T16:04:56.96-0500 [App/0]      OUT [WEB] 2015/06/24 21:04:56 Failed to deliver message due to SMTP error: x509: certificate is valid for mail.xyz.com, not relaymail.xyz.com
2015-06-24T16:04:56.96-0500 [App/0]      OUT [WEB] 2015/06/24 21:04:56 Message failed to send, retrying at: Jun 24, 2015 at 9:05pm (UTC)
2015-06-24T16:04:56.96-0500 [App/0]      OUT [METRIC] {"kind":"counter","payload":{"name":"notifications.worker.retry"}}

From the above logs, it appears that the smtp is looking for a valid SSL certificate, however from the env variable for notifications app 'VERIFY_SSL' is set to false. Checking the env variables for 'notifications' app reveals the following:

-> cf env notifications

--snip--
ser-Provided:
CC_HOST: https://api.system.abc.xyz.com
DATABASE_URL: mysql://f0665babee1b5a1e6a88:[email protected]:3306/notifications
DOMAIN: system.abc.xyz.com
ENCRYPTION_KEY: a1a777d2ea66476fe99d
GOBBLE_MIGRATIONS_DIR: $HOME/gobble/migrations
ROOT_PATH: $HOME
SENDER: [email protected]
SMTP_AUTH_MECHANISM: none
SMTP_CRAMMD5_SECRET:
SMTP_HOST: smtp.sendgrid.net
SMTP_PASS: abc.xyz
SMTP_PORT: 587
SMTP_TLS: true
SMTP_USER: krashpanic
UAA_CLIENT_ID: notifications
UAA_CLIENT_SECRET: fbb72bce255561f08930
UAA_HOST: https://uaa.system.abc.xyz.com
VERIFY_SSL: false

Environment

Cause

This is a bug in the notification app where the SMTP client ignores the "VERIFY_SSL" set to "false" environment variable and enforces having a valid SSL certificate.

Resolution

This is a known issue in PCF v1.4, the fix is planned to be available in PCF v1.5.1 release.

If you are adding new users via the app-manager/console, the workaround is to manually send the registration link to the user. The other fix is to provide a valid SSL certificate on the SMTP server.