Avi Load Balancer Controller Crash During SE PCAP Collection with DEBUG_PCAP_ALL Flag enable

Products

• VMware Avi Load Balancer

Issue/Introduction

Avi ALB Controllers running versions 22.1.x, 30.1.x, and 30.2.x may experience a crash when collecting Service Engine (SE) packet captures (PCAP) with the DEBUG_PCAP_ALL flag. This crash occurs on controllers when ServiceEngine configured with more than 9 CPU cores.

StackTrace: Core Archive File from Controller node

Command line: /opt/avi/scripts/debugse_callback.py -o {"flags":[{"flag":"DEBUG_PCAP_ALL"}],"uuid":"se-uuid"} -n {"uuid":"se-uuid"}
Traceback (most recent call last):
  File "/opt/avi/scripts/debugse_callback.py", line 48, in <module>
    main()
  File "/opt/avi/scripts/debugse_callback.py", line 45, in main
    debug_se.collect_se_pcap(new_debugse_obj, old_debugse_obj)
  File "/opt/avi/python/lib/avi/rest/callbacks_custom.py", line 3591, in collect_se_pcap
    return self.copy_se_file_new_se_version(se_uuid, pcap_types,old_pb,model_obj)
  File "/opt/avi/python/lib/avi/rest/callbacks_custom.py", line 3419, in copy_se_file_new_se_version
    idx = self._get_the_least_recent_pcap_file(files[core_num])
  File "/opt/avi/python/lib/avi/rest/callbacks_custom.py", line 3200, in _get_the_least_recent_pcap_file
    with open(SE_PCAP_DIR  + os.path.basename(_file), 'rb+') as f1:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/avi/se-pcap/se-uuid_se_core_10_part1.pcap'

Identify the Crash from Controller UI

Navigate to Controller UI > Administration > Support > Crash Reports > Check for process crash: debugse_callback.py

"core_archive": "core_archive.XXXXXX.tar.gz", 
"reason": "debugse_callback.py on node (X-X-X-X) crash -- Pls investigate further. core_archive.XXXXXX.tar.gz generated",
"process_name": "debugse_callback.py",

 

 

 

Environment

 

• Avi ALB versions: 22.1.x, 30.1.x, 30.2.x
• ServiceEngine with more than 9 CPU cores.
• SE PCAP collection initiated with the DEBUG_PCAP_ALL flag
  

  Steps to enable debug pcap all on SE from Shell
  
  [admin:]: > debug serviceengine Avi-se-xxxx
  [admin:]: debugserviceengine> flags flag debug_pcap_all
  [admin:]: debugserviceengine:flags> save
  [admin:]: debugserviceengine> save

 

Cause

• An incorrect regular expression within the internal file structure caused PCAP files for cores 10 through 19 to be incorrectly matched with files from core 1 and with themselves. This resulted in the files for cores 10-19 being read, updated, or deleted as part of core 1's processing. Subsequent attempts to access these already removed files led to a controller crash.
• This issue specifically impacts Packet capture collection using the DEBUG_PCAP_ALL flag on Service Engine with debug serviceengine <SE-Name> mode.

Resolution

Upgrade to one of the following fixed versions:

• 22.1.7-2p6
• 30.2.3
• 31.2.1
• 31.1.2