Controller Debug Log Collection Fails When Follower Node is Down

Products

• VMware Avi Load Balancer

Issue/Introduction

• When operating with three-node Avi Controller cluster, you may encounter failures when attempting to collect debug logs or generate tech support bundles if one of the follower nodes is offline or inaccessible. In such scenarios, the collected debug logs will have a size of 0 bytes, indicating a failed collection process.

• The tech_support.log file will show errors related to tasks failing on the unreachable follower node and file copy operations failing.
  
  

  root@:/# cd /var/lib/avi/log/
  root@:/var/lib/avi/log# vim tech_support.log
  
  INFO [tech_support.preprocess:232] START: Techsupport preprocess
  Traceback (most recent call last):
    File "", line 80, in ignored
      yield
    File "", line 215, in _create_odir
      cls.status_file = init_techsupport_status(cls.txId, **cls.kwargs)
  avi.util.exceptions.FileUploadError: Could not copy /path/file.status to node.controller.local, stdout=b'', stderr=b'', exitcode=9^[[0m
  INFO [tech_support.preprocess:247] OutputDir[None]
  INFO [tech_support.preprocess:248] END: Techsupport preprocess

Environment

This issue occurs in Avi Controller clusters with:

• A three-node cluster deployment.
• One of the follower nodes is currently down or unreachable during the debug log or tech support collection attempt.

Affects Version(s):

22.1.x
30.1.x
30.2.1, 30.2.2, 30.2.3
31.1.1

 

Cause

• During the Controller debug log / tech support collection process in a clustered environment, the system attempts to replicate status files and coordinate tasks across all active Controller nodes, including followers. When a follower node is down, the system encounters an error while trying to execute remote tasks or copy necessary files (like status files) to that unreachable node.
• This failure in inter-node communication or file synchronization prevents the complete and successful generation of the debug logs or tech support bundle, leading to a 0-byte output file.

Resolution

Please upgrade the system to the fix version.

AV-238480: Controller debug log collection fails when follower node is down
Fix Version(s): 30.2.4, 31.1.2, 31.2.1

Workaround:

Use the generate_techsupport.py script to collect tech support from individual controller node

SSH into Active Controller Nodes

admin@:~$ sudo -i
root@:~#
root@:~# cd /opt/avi/scripts/
root@:/opt/avi/scripts# python3 generate_techsupport.py
Started Upgrade Analyzer on node: CONTROLLER
Final Tech Support file created in:/var/lib/avi/tech_support/debuglogs-YYYYMMDD.tar.gz