"Response exceeded maximum allowed length" error after running Isolation Segment smoke test in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The errand, smoke_tests_isolation, is introduced in Isolation Segment v2.9 and above.  

It may fail with the error, "Response exceeded maximum allowed length".

Task 5324 | 08:04:44 | Preparing deployment: Preparing deployment
Task 5324 | 08:04:45 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task 5324 | 08:04:50 | Preparing deployment: Preparing deployment (00:00:06)
Task 5324 | 08:04:50 | Running errand: isolated_diego_cell/05444c81-f2c3-49ee-1ef4-c624561b2133 (14) (00:14:02)
L Error: Response exceeded maximum allowed length
Task 5324 | 08:18:52 | Error: Response exceeded maximum allowed length

Environment

Product Version: 2.9

Resolution

The errand is executed on one isolated_diego_cell VM.

It firstly pushes one test app to the Isolation Segment, with a domain can be specified at Ops Manager > Isolation Segment > Settings > Smoke Test > Domain. Then the errand will try to reach out to the app over HTTPS. 

In the case, the errand can not successfully connect to the test app over HTTPS and as a result it will keep repeating the error. For example:

...
HTTP Request: https://****.<APP_DOMAIN>/json
Instance #: 0
Error: Get https://****.<APP_DOMAIN>/json: dial tcp 10.100.100.5:443: connect: connection refused

HTTP Request: https://****.<APP_DOMAIN>/json 
Instance #: 0 
Error: Get https://****.<APP_DOMAIN>/json: dial tcp 10.100.100.5:443: connect: connection refused
...

The repeated error messages finally results in very large response body. Once it exceeds the limit that BOSH agent can handle, the BOSH agent returns the error "Response exceeded maximum allowed length" without further details. 

To workaround this issue, follow these steps: 

1. Collect the isolated_diego_cell VM logs for the VM the errand was executed on.

2. Review smoke_tests_isolation/smoke_tests_isolation.stdout.log.

3. Based on the information recorded in the smoke test log, take the necessary action. For the example in this article, the error is caused by a networking connection issue, which is usually due to misconfigured network.