App push hangs and times out post TAS upgrade to TAS v2.11.14+ as ASG becomes more strict due to using Golang v1.17

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Customer is facing the issue that apps are failed to deploy when upgrading to TAS v2.11.14+. In the app log, you see a post cf push staging stage completed, however, the actual app container never gets created and started. This is followed by app timeout and the app fails to push.

Uploading complete
Cell xxxxx stopping instance xxxx
Cell xxxx destroying container for instance xxxx
Cell xxxx successfully destroyed container for instance xxxx

Waiting for app to start...
Start app timeout

TIP: Application must be listening on the right port. Instead of hard coding the port, use the $PORT environment variable.

Use 'cf logs bam-suitability-service --recent' for more information

FAILED

There is no error message in the app logs or in the Diego Cell logs. Bulletin Board System (BBS) in the Diego Database VMs reports in bbs.stdout.log an error message similar to the following:

{"timestamp":"2023-03-06T11:38:02.895296975Z","level":"error","source":"bbs","message":"bbs.request.desire-lrp.invalid-request","data":{"error":"Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: destinations [ Invalid IP ], Invalid field: egress_rules, Invalid field: -(truncated)","session":"219968938.1"}}

Environment

Product Version: 2.11

Resolution

At first, we are discussing the situation that no changes have been made to ASG before/after TAS upgrade. If you have created or modified ASG, please refer to this KB to review if an invalid syntax of one or more of the IP, IP Range, Port, or Protocol fields for Application Security Group.

After confirmed that there is no invalid syntax in ASG, please check if there is any IP address with a leading zero in a security group bound to the platform-wide running or staging security groups. This was due to a bump in Golang v1.17 that had stricter checking for IP’s with leading 0's, basically Golang v1.17 would drop these now. For example:

[
  {
    "protocol": "tcp",
    "destination": "10.00.00.192/26",
    "ports": "80",
    "log": false,
    "description": "ASG validate"
  }
]

There was a Feature Improvement starting from TAS v2.11.14 where we would check if IP’s were compliant and if they had leading 0's we would rewrite the IP in Bosh Templates in the DB to drop the leading Zero.

[Feature Improvement] Golang v1.17 contains stricter IP parsing standards, so IP addresses with leading zeros in any octets cause a BOSH template failure. Operators can remove the leading zeros and try deploying again.

This requires user intervention to update the ASG removing the leading zeros, then app will up and running.