Diego service modules crash if monitoring them by Dynatrace OneAgent with Infrastructure mode

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Diego service modules such as BBS, Auctioneer, and so on frequently crash with the following Goroutine runtime error if monitoring them by Dynatrace OneAgent with Infrastructure mode:

runtime: unexpected return pc for net/http.serverHandler.ServeHTTP called from 0x11ea0
stack: frame={sp:0xc00242dc18, fp:0xc00242dc48} stack=[0xc00242a000,0xc00242e000)
000000c00242db18:  000000c00242db48  0000000000c9d567 <code.cloudfoundry.org/bbs/handlers.(*UnavailableHandler).ServeHTTP+119> 
000000c00242db28:  000000c0000104b8  000000000100e720 
000000c00242db38:  000000c00073b340  000000c002cf6a00 
000000c00242db48:  000000c00242db78  0000000000c9f0cd <code.cloudfoundry.org/bbs/handlers.UnavailableWrap.func1+77> 
000000c00242db58:  000000c0003c01c0  000000000100e720 
000000c00242db68:  000000c00073b340  000000c002cf6a00 
000000c00242db78:  000000c00242dba0  00000000006ed974 <net/http.HandlerFunc.ServeHTTP+68> 
000000c00242db88:  000000000100e720  000000c00073b340 
000000c00242db98:  000000c002cf6a00  000000c00242dbe0 
000000c00242dba8:  0000000000c8a41c <code.cloudfoundry.org/bbs/handlers/middleware.RecordRequestCount.func1+124>  000000c0003c01e0 
000000c00242dbb8:  000000000100e720  000000c00073b340 
000000c00242dbc8:  000000c002cf6a00  0000000000ff8260 
000000c00242dbd8:  000000c0003c01e0  000000c00242dc08 
000000c00242dcd8:  0000000000000000  000000c0037ef380 
  :
000000c00242dce8:  000000c00068e481  000000000000000c 
000000c00242dcf8:  000000000000000c  0000000000e50780 
000000c00242dd08:  0000000000000005  000000c001174958 
000000c00242dd18:  0000000001011420  0000000000000000 
000000c00242dd28:  000000c001174700  0000000000ff71c0 
000000c00242dd38:  000000c000a8dc28  0000000000000005 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0xe88094, 0x11)
	/var/vcap/data/packages/golang-1-linux/3709f0468acaed579a9afde694602831eb5e9e44/src/runtime/panic.go:774 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000fef380, 0x0, 0x0, 0x7fffffff, 0xebf5f0, 0x7fb1b7ffebe0, 0x0, ...)
  :

To make matters worse, those crashed modules are not failed over to standby instances and the Diego system doesn't work at all once this crash issue happens.

Environment

Product Version: 2.9

Resolution

This is derived from a bug in Dynatrace OneAgent which omits protection mechanisms against side effect to cause this kind of crash if Infrastructure mode is selected with Dynatrace OneAgent.

The quick workaround is to select Full stack mode rather than Infrastructure mode.

This issue has been resolved within the following versions of Dynatrace OneAgent. Ask Dynatrace support if you need those fixes:
 

• 1.199.11.20200720-182244
• 1.197.124.20200720-181337
• 1.195.161.20200720-160625
• 1.193.201.20200720-193623