Unable to access Pivotal Cloud Foundry logs in AWS

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

The following issue arises when trying to access the PCF logs through the CF CLI utility in AWS environments: "Error dialing traffic controller server: websocket: bad handshake".

The Loggregator is responsible for streaming logs and metrics from all user apps and system components from Pivotal Application Service.

These logs are made available via the Loggregator's primary transport mechanism called Doppler. Logs are streamed from the Loggregator's traffic controller using the HTTP WebSocket Protocol (WSS).  

This article explains how to alleviate this issue when accessing PCF logs via the CF CLI.

Environment

Cause

When running "cf logs", you are connecting to the application logging subsystem called Loggregator. The error shown above occurs when this connection fails.

Resolution

In order to see what your Doppler logging endpoint is set to currently, please run the following command.

$ cf curl /v2/info | jq .doppler_logging_endpoint
 wss://doppler.###.#.#.##.xip.io:4443

By default, in AWS the Loggregator is listening on TCP port 4443. In order to connect to this port and retrieve logs, you will need to allow the ingress of traffic on TCP port 4443 to your PCF environment.

In the EC2 Dashboard, select Security Groups, check if the security group PCF_ELB_SecurityGroup exists and has the following rule in place (4.); if not please add it as follows.

Log into your AWS console and add the following security group:

1. On the EC2 Dashboard, go to, Security Groups > Create Security Group.
2. Enter a security group name and description: PCF_ELB_SecurityGroup.
3. Select the VPC to which to deploy the ELB.
4. Click the "Inbound" tab and add rules to allow traffic to port 4443 from 0.0.0.0/0

This allows you access to connect to Loggregator via the WebSocket Protocol which provides access to the logs.