"Apply Changes" fails with TAS notifications errand

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When notifications errand is run during "Apply Changes", it fails with starting the deployed notification app.

===== 2023-10-17 16:10:58 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=x.x.x.x --deployment=cf-0620a1e1a5a0b42e2b39 run-errand deploy-notifications --instance clock_global/
first"
Using environment 'x.x.x.x' as client 'ops_manager'

Using deployment 'cf-0620a1e1a5a0b42e2b39'

Task 27168

Task 27168 | 16:10:58 | Preparing deployment: Preparing deployment (00:00:08)
Task 27168 | 16:11:06 | Running errand: clock_global/19b76ef2-0742-45d5-b55c-909440b6ea28 (1) (00:02:06)
Errand 'deploy-notifications' completed with error (exit code 1)
Task 27168 | 16:13:12 | Fetching logs for clock_global/19b76ef2-0742-45d5-b55c-909440b6ea28 (1): Finding and packing log files (00:00:02)

Task 27168 Started  Tue Oct 17 16:10:58 UTC 2023
Task 27168 Finished Tue Oct 17 16:13:14 UTC 2023
Exit code 1
......
           Starting app notifications in org system / space notifications-with-ui as notifications...  
           Downloading go_buildpack...  
......
           0 of 1 instances running, 1 crashed  
           Deleting app notifications in org system / space notifications-with-ui as notifications...  
           OK  
           Renaming app notifications-venerable to notifications in org system / space notifications-with-ui as notifications...  
           OK  
           error: Start unsuccessful  
             

The notifications app logs contain more errors about startup failure.

   2023-10-19T03:07:07.50+0000 [CELL/1] OUT Cell cd77ae65-cc7b-4679-b494-386b28f374f3 creating container for instance 5a2628a5-befd-4aca-4ff6-c1d1
   2023-10-19T03:07:07.50+0000 [CELL/2] OUT Cell d11a5cff-26d5-4dd8-8c36-e97130bc7e3e creating container for instance f85fa5f9-a6a6-4e81-7225-5d93
   2023-10-19T03:07:07.80+0000 [CELL/1] OUT Cell cd77ae65-cc7b-4679-b494-386b28f374f3 successfully created container for instance 5a2628a5-befd-4aca-4ff6-c1d1
   2023-10-19T03:07:07.83+0000 [CELL/2] OUT Cell d11a5cff-26d5-4dd8-8c36-e97130bc7e3e successfully created container for instance f85fa5f9-a6a6-4e81-7225-5d93
   2023-10-19T03:07:08.31+0000 [CELL/2] OUT Downloading droplet...
   2023-10-19T03:07:08.42+0000 [CELL/2] OUT Downloaded droplet
   2023-10-19T03:07:08.48+0000 [CELL/1] OUT Downloading droplet...
   2023-10-19T03:07:08.51+0000 [CELL/2] OUT Starting health monitoring of container
   2023-10-19T03:07:08.60+0000 [CELL/1] OUT Downloaded droplet
   2023-10-19T03:07:08.67+0000 [CELL/1] OUT Starting health monitoring of container
   2023-10-19T03:07:23.73+0000 [APP/PROC/WEB/2] OUT {"timestamp":"1697684843.731485844","source":"notifications","message":"notifications.smtp-connect-errored","log_level":3,"data":{"error":"server timeout","trace":"goroutine 1 [running]:\ngithub.com/pivotal-golang/lager.(*logger).Fatal(0xc000202180, {0x55c5b206cb52, 0x14}, {0x55c5b22f9da0?, 0xc000025310}, {0x0, 0x0, 0xc00019a000?})\n\t/tmp/app/vendor/github.com/pivotal-golang/lager/logger.go:131 +0x1fc\ngithub.com/cloudfoundry-incubator/notifications/application.Application.VerifySMTPConfiguration({{{0xc00002a068, 0x25}, {0x55c5b223c014, 0x1}, 0x0, 0x3, {0xc000034200, 0x74}, {0xc000038013, 0xa7}, ...}, ...})\n\t/tmp/app/application/application.go:90 +0x251\ngithub.com/cloudfoundry-incubator/notifications/application.Application.Run({{{0xc00002a068, 0x25}, {0x55c5b223c014, 0x1}, 0x0, 0x3, {0xc000034200, 0x74}, {0xc000038013, 0xa7}, ...}, ...})\n\t/tmp/app/application/application.go:60 +0x5a\nmain.main()\n\t/tmp/app/main.go:19 +0x2a5\n"}}

The error "notifications.smtp-connect-errored" indicates issue with connecting the configured SMTP server. 

A manual test with command nc -vz <smtp server IP> 25 on Ops Manager VM also fails, which indicates the network problem with SMTP server IP/PORT (might be blocked by firewall).

Environment

Product Version: 2.11

Resolution

The connectivity issue between TAS and SMTP server needs to be fixed first by related team. The SMTP server and port fields can't be left blank. Otherwise the notifications errand will fail with error as shown below.