Rep fails to start with Diego v2.61.0+ with Dynatrace oneagent < v1.237

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This KB is to highlight an incompatibility observed between Dynatrace oneagent < v1.237 and Golang v1.17.8.

Note: This behavior has only been observed on non-vSphere environments.

Diego v2.61.0 is compiled with Golang 1.17.8. If a Dynatrace oneagent < v1.237 is applied to a bosh deployment that contains releases from Diego v2.61.0 then it is possible there may be failures as the versions are incompatible.

Exposed Tile Versions

• TAS 2.7.48+, 2.10.28+, 2.11.16+, 2.12.9+
• Isolation Segment 2.7.43+, 2.10.23+, 2.11.12+, 2.12.6+
• TASW 2.7.39+, 2.10.23+, 2.11.12+, 2.12.6+

As an example this log snippet was seen from the rep process on an Azure environment running Diego v2.61.0 and Dynatrace oneagent v1.235.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x42571e]

goroutine 3 [running]:
runtime.throw({0x1011bbf, 0xc000063630})
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/panic.go:1198 +0x71 fp=0xc000063618 sp=0xc0000635e8 pc=0x4370b1
runtime.sigpanic()
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/signal_unix.go:719 +0x396 fp=0xc000063668 sp=0xc000063618 pc=0x44d696
runtime.(*sweepLocked).sweep(0x1aa37a0, 0x0)
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/mgcsweep.go:444 +0xa5e fp=0xc000063740 sp=0xc000063668 pc=0x42571e
runtime.sweepone()
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/mgcsweep.go:290 +0x109 fp=0xc0000637b0 sp=0xc000063740 pc=0x4246e9
runtime.bgsweep()
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/mgcsweep.go:166 +0x105 fp=0xc0000637e0 sp=0xc0000637b0 pc=0x4245a5
runtime.goexit()
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000637e8 sp=0xc0000637e0 pc=0x46a9c1
created by runtime.gcenable
	/var/vcap/data/packages/golang-1-linux/941635974642291bcce3b36cba6a9bea66d05ff2/src/runtime/mgc.go:181 +0x55

Environment

Product Version: 2.10

Resolution

Workaround 1 (preferred)
Update to a Dynatrace oneagent version that's compatible with Golang 1.17.8

• Update to Dynatrace oneagent >= v1.237.
• This is the agent downloadurl section from this doc.
• If downloadurl is omitted or specifies the latest, then you don't have to update the Dynatrace runtime config. If you have a version specified then you have to update the Dynatrace runtime config and apply changes.
• If you have a Diego cell in this condition simply run the pre-start script in /var/vcap/jobs/dynatrace-oneagent/bin/pre-start to download the new agent version to the Diego cell. Once the new agent is installed, the rep process can be restarted and should be successful.

Workaround 2
Disable rep monitoring in Dynatrace dashboard

1. Log into your Dynatrace dashboard
2. Go to Manage -> Settings -> Processes and containers -> Built-in process monitoring rules
3. Disable this rule: Do monitor processes if EXE name equals 'rep'
4. Save changes
5. Run monit restart rep on any failing Diego cells