CARR script freezes during certificate regeneration in VMware NSX

Products

• VMware NSX

Issue/Introduction

• The CARR python script hangs or stops progressing after entering node credentials.
• NSX Manager memory utilization is observed at or above 90%.
• NSX Manager uptime exceeds 365 days.
• No specific error message is generated; the script simply remains idle after its executed.

Environment

• VMware Cloud Foundation (VCF)
• VMware NSX 4.1.x

Cause

High memory utilization and significant appliance uptime lead to resource exhaustion on NSX Manager nodes. This prevents the python script from initiating remote inventory tasks or processing the certificate store.

Resolution

1. Terminate the current script execution using Ctrl+C.
2. Perform a rolling reboot of the NSX Manager cluster nodes to clear cached memory and reset system resources (refer to KB 375204 for rolling reboot guidance).
3. Connect to each NSX Manager via SSH and verify memory usage has returned to normal levels using get processes monitor
4. Rerun the CARR script starting with the mandatory dry run: ./start.sh -d.

Support: If the script continues to hang after a reboot, please contact Broadcom Support and provide the carr.log file.