Accessing apps failed with "502 Bad Gateway" due to time on diego_cell out of synchronization
search cancel

Accessing apps failed with "502 Bad Gateway" due to time on diego_cell out of synchronization

book

Article ID: 297999

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Accessing some apps (including "Apps Manager") returned a "502 Bad Gateway" error. And "x509: certificate has expired or is not yet valid" was seen in gorouter logs. 
{"log_level":3,"timestamp":1580785407.785722,"message":"backend-endpoint-failed","source":"vcap.gorouter","data":{"route-endpoint":{"ApplicationId":"85fa4540-e335-42c5-8a85-8d161268 b955","Addr":"10.0.0.1:61062","Tags":{"app_id":"85fa4540-e335-42c5-8a85-8d161268b955","app_name":"apps-manager-js-blue","component":"route-emitter","instance_id":"1","organizati on_id":"74e90b47-24b9-44d5-a0a0-314c73941fe6","organization_name":"system","process_id":"85fa4540-e335-42c5-8a85-8d161268b955","process_instance_id":"cae415bc-bc03-4147-67bd-2400"," process_type":"web","source_id":"85fa4540-e335-42c5-8a85-8d161268b955","space_id":"22d5e4fe-5b26-4337-82ce-8b5faf5c0fa5","space_name":"system"},"RouteServiceUrl":""},"error":"x509: certificate has expired or is not yet valid","attempt":1,"vcap_request_id":"8dcad572-7e71-4d09-7348-1226bfedca38"}}
Checked certificate returned from one of "Apps Manger" instances in question and found time on diego_cell VM hosting this instance was beyond the validity period of the certificate. 
Validity
     Not Before: Feb  4 14:23:52 2020 GMT
     Not After : Feb  5 14:23:52 2020 GMT

diego_cell/6a8ce2e1-f933-4780-824d-280a30365404: stdout | Tue Feb  4 03:45:59 UTC 2020
An incorrect time issue existed on several diego_cell VMs: 
~$ bosh -d cf-xxxx ssh diego_cell -c 'date'|grep "UTC 2020"
diego_cell/b38f9f1d-39db-44e4-8075-b4df8cbc05ed: stdout | Tue Feb  4 14:59:53 UTC 2020
diego_cell/d7c9cf1d-3fd6-4374-9046-82f41fa5da6b: stdout | Tue Feb  4 15:00:07 UTC 2020
diego_cell/1a9afc88-1439-4df6-b4d4-4102af633c6b: stdout | Tue Feb  4 14:59:49 UTC 2020
diego_cell/7f4178d0-d6b4-4569-b076-cbc152810f33: stdout | Tue Feb  4 15:00:13 UTC 2020
diego_cell/440a26bc-b2d4-4397-901d-4c3ad19eeb44: stdout | Tue Feb  4 14:59:46 UTC 2020
diego_cell/6a8ce2e1-f933-4780-824d-280a30365404: stdout | Tue Feb  4 03:45:59 UTC 2020
diego_cell/2446e6b4-532a-47f9-a34b-1a203fe3147c: stdout | Tue Feb  4 14:59:40 UTC 2020
diego_cell/75987c78-ad9c-40fa-8317-0a5379e18c20: stdout | Tue Feb  4 03:45:59 UTC 2020
diego_cell/8280c1c1-b69b-4d9b-b773-a1b36a6727fe: stdout | Tue Feb  4 15:10:19 UTC 2020
diego_cell/7122da00-4aee-4cc5-bf32-9363fb9f615c: stdout | Tue Feb  4 15:11:13 UTC 2020
It seemed diego_cell VMs were not synchronizing time from NTP servers in a timely manner. Last modified time of "sync-time.out" file was long time ago: 
~$ bosh -d cf-xxxx ssh diego_cell -c 'sudo ls -l /var/vcap/bosh/log/sync-time.out' |grep sync-time
diego_cell/d7c9cf1d-3fd6-4374-9046-82f41fa5da6b: stdout | -rw-r--r-- 1 root root 134 Jan 17 10:03 /var/vcap/bosh/log/sync-time.out
diego_cell/b38f9f1d-39db-44e4-8075-b4df8cbc05ed: stdout | -rw-r--r-- 1 root root 134 Jan  9 11:07 /var/vcap/bosh/log/sync-time.out
diego_cell/1a9afc88-1439-4df6-b4d4-4102af633c6b: stdout | -rw-r--r-- 1 root root 134 Jan  9 09:41 /var/vcap/bosh/log/sync-time.out
diego_cell/440a26bc-b2d4-4397-901d-4c3ad19eeb44: stdout | -rw-r--r-- 1 root root 134 Jan 17 10:19 /var/vcap/bosh/log/sync-time.out
diego_cell/75987c78-ad9c-40fa-8317-0a5379e18c20: stdout | -rw-r--r-- 1 root root 134 Jan  9 10:35 /var/vcap/bosh/log/sync-time.out
diego_cell/8280c1c1-b69b-4d9b-b773-a1b36a6727fe: stdout | -rw-r--r-- 1 root root 133 Jan  9 10:57 /var/vcap/bosh/log/sync-time.out
diego_cell/7f4178d0-d6b4-4569-b076-cbc152810f33: stdout | -rw-r--r-- 1 root root 133 Jan  9 10:47 /var/vcap/bosh/log/sync-time.out
diego_cell/7122da00-4aee-4cc5-bf32-9363fb9f615c: stdout | -rw-r--r-- 1 root root 134 Jan 17 10:14 /var/vcap/bosh/log/sync-time.out
diego_cell/6a8ce2e1-f933-4780-824d-280a30365404: stdout | -rw-r--r-- 1 root root 134 Jan 17 10:09 /var/vcap/bosh/log/sync-time.out
diego_cell/2446e6b4-532a-47f9-a34b-1a203fe3147c: stdout | -rw-r--r-- 1 root root 134 Jan  9 09:52 /var/vcap/bosh/log/sync-time.out


Environment

Product Version: 2.7

Resolution

Restart chrony service on all diego_cell VMs, time on diego_cell VMs was updated and synchronized with NTP server.  
~$ bosh -d cf-xxxx ssh diego_cell -c 'sudo systemctl restart chrony'
~$ bosh -d cf-xxxx ssh diego_cell -c 'date' |grep UTC
diego_cell/b38f9f1d-39db-44e4-8075-b4df8cbc05ed: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/d7c9cf1d-3fd6-4374-9046-82f41fa5da6b: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/1a9afc88-1439-4df6-b4d4-4102af633c6b: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/440a26bc-b2d4-4397-901d-4c3ad19eeb44: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/75987c78-ad9c-40fa-8317-0a5379e18c20: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/7f4178d0-d6b4-4569-b076-cbc152810f33: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/6a8ce2e1-f933-4780-824d-280a30365404: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/8280c1c1-b69b-4d9b-b773-a1b36a6727fe: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/2446e6b4-532a-47f9-a34b-1a203fe3147c: stdout | Tue Feb  4 04:19:50 UTC 2020
diego_cell/7122da00-4aee-4cc5-bf32-9363fb9f615c: stdout | Tue Feb  4 04:19:50 UTC 2020
Then restart affected applications to renew their certificates.

Powered by Wolken Software