Service Engines may fail after upgrade from 22.1.x to 30.2.1

Products

• VMware Avi Load Balancer

Issue/Introduction

Service Engine may fail after upgrade from 22.1.x to 30.2.1 when Client Insights is enabled on the Virtual Services.

Below is an example of stack trace from such a Service Engine failure. This stack trace would be found by navigating to Administration>Support>Crash Reports.

[New LWP 2683]
[New LWP 2718]
[New LWP 2717]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `se_dp: worker process 0:                          '.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f2a02d7100b in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f29ff18cf40 (LWP 2683))]
#0  0x00007f2a02d7100b in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f2a02d50859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f2a02dbb26e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f2a02dc32fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f2a02dc60b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f2a02dc8299 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00005574a27ceb3c in se_malloc (size=4104, size@entry=<error reading variable: value has been optimized out>, type=<optimized out>, type@entry=<error reading variable: value has been optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/utils/se_malloc.c:1608
#7  0x00005574a24f6318 in ngx_real_alloc (size=size@entry=4096, log=0x5574a7193f78, mtype=SE_MTYPE_L7_REQUEST_MEMPOOL, alloc_func=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_alloc.c:26
#8  0x00005574a24f6490 in ngx_alloc (size=size@entry=4096, log=<optimized out>, mtype=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_alloc.c:43
#9  0x00005574a22e4a77 in ngx_palloc_large (size=4096, pool=0x5574a7e66360) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/core/ngx_palloc.c:352
#10 ngx_palloc (pool=pool@entry=0x5574a7e66360, size=size@entry=4096) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/core/ngx_palloc.c:233
#11 0x00005574a22e7e4b in ngx_create_temp_buf (pool=0x5574a7e66360, size=4096) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/core/ngx_buf.c:52
#12 0x00005574a23ce4ce in ngx_http_gzip_filter_get_buf (ctx=0x5574a7093ac0, r=0x5574a7e663d0) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_gzip_filter_module.c:841
#13 ngx_http_gzip_body_filter (in=<optimized out>, r=0x5574a7e663d0) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_gzip_filter_module.c:505
#14 ngx_http_gzip_body_filter (r=0x5574a7e663d0, in=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_gzip_filter_module.c:394
#15 0x00005574a247fe5e in ngx_http_csrf_body_filter (r=<optimized out>, in=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_csrf_filter_module.c:627
#16 0x00005574a247d499 in ngx_http_rum_body_filter (r=0x5574a7e663d0, in=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_rum_filter_module.c:563
#17 0x00005574a23d14c6 in ngx_http_gunzip_body_filter (in=<optimized out>, r=0x5574a7e663d0) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_gunzip_filter_module.c:306
#18 ngx_http_gunzip_body_filter (r=0x5574a7e663d0, in=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/ngx_http_gunzip_filter_module.c:203
#19 0x00005574a24d6e9c in ngx_http_lua_body_filter (in=<optimized out>, r=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/lua/ngx_http_lua_bodyfilterby.c:315
#20 ngx_http_lua_body_filter (r=0x5574a7e663d0, in=0x5574a7093f60) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/modules/lua/ngx_http_lua_bodyfilterby.c:223
#21 0x00005574a238f6fa in ngx_http_output_filter (r=r@entry=0x5574a7e663d0, in=0x5574a7093f60) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/ngx_http_core_module.c:2285
#22 0x00005574a2430996 in ngx_http_upstream_process_non_buffered_request (r=0x5574a7e663d0, do_write=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/ngx_http_upstream.c:5373
#23 0x00005574a2427b9c in ngx_http_upstream_handler (ev=0x5574a7da28a8) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/http/ngx_http_upstream.c:1246
#24 0x00005574a23843ba in ngx_ipstk_dispatch_event (event=0x600047bad000) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/event/modules/ngx_ipstk_module.c:341
#25 0x00005574a27cdc07 in ipstk_msg_dispatch () at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/utils/ipstk_api.c:544
#26 0x00005574a2384a30 in ngx_ipstk_process_events (cycle=0x5574a66a3780, timer=<optimized out>, flags=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/event/modules/ngx_ipstk_module.c:579
#27 0x00005574a24fc7e8 in ngx_worker_process_cycle (cycle=cycle@entry=0x5574a66a3780, data=data@entry=0x7ffd868c5ce8) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_process_cycle.c:970
#28 0x00005574a24f9c02 in ngx_spawn_process (cycle=cycle@entry=0x5574a66a3780, proc=proc@entry=0x5574a24fc4d0 <ngx_worker_process_cycle>, data=data@entry=0x7ffd868c5ce8, name=name@entry=0x5574a2bea0ab "worker process", respawn=respawn@entry=-3) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_process.c:216
#29 0x00005574a24fd774 in ngx_start_worker_processes (cycle=cycle@entry=0x5574a66a3780, n=2, type=type@entry=-3) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_process_cycle.c:467
#30 0x00005574a24fe6ba in ngx_master_process_cycle (cycle=0x5574a66a3780) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/os/unix/ngx_process_cycle.c:155
#31 0x00005574a22df3c3 in main (argc=<optimized out>, argv=<optimized out>) at /home/aviuser/workspace/30.2.1-2p2-Release/service_engine/nginx/src/core/nginx.c:336

 

 

 

 

Resolution

The issue is fixed in 30.2.2 release. 

If upgrading to 30.2.1 release, it is recommended to disable client insights on virtual services prior to the upgrade to avoid running into this issue.