Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX

Products

• VMware NSX

Issue/Introduction

This script is intended to be used to resolve certificate management issues on NSX 3.2.x and 4.x. It performs integrity checks and recovery operations for NSX self-signed certificates.

Environment

VMware NSX 4.x
VMware NSX-T Data Center 3.2.x

Resolution

The script will make an assessment of certificate remediation needed, present the proposed changes and ask for approval to proceed.

Client Requirements:

• For NSX 4.1.x and 4.2.x, the script can be run directly on a Local or Global NSX Manager from /root directory 
• For NSX 3.2.x, 4.0.x an external client machine is required meeting the following requirements
  

  Python version requirements

  • 3.13+ requires the client machine to have internet connectivity and cannot be run offline
  • 3.8 to 3.12 can run with or without client internet connectivity
    
    

  OS : MAC and Linux

  Architecture -  (if internet connection is there then there is no restriction, dependencies are downloaded)

  • MAC : : "macosx_10_9_x86_64" "macosx_11_0_arm64"
  • Linux : : "musllinux_1_1_x86_64" "musllinux_1_1_aarch64" "manylinux_2_17_x86_64" "manylinux_2_17_s390x" "manylinux_2_17_aarch64" 
    
    
• In all cases, the script requires the following ports to be open between the client machine and the 3 NSX Managers
  • ssh port 22
  • https port 443
  • corfu port 9000
    Note: if running on the NSX Manager directly, ports 443 and 9000 will already be open between the 3 Managers.
    
    
• ssh access via admin and root users must be enabled on the NSX Manager, if needed see Enable ssh root access for NSX appliances.
  
  

Execution Notes:

• This script can be ran on any node and it will reach out to the respective NSX nodes in the correct order.
• Ensure that you have a recent, valid backup of your NSX managers. Ensure that you know the passphrase for your backups.
• On NSX 4.1.x and 4.2.x, the script should be run from the /root directory, it will not work from the /tmp directory
• admin and root passwords of NSX Manager are required as inputs.
• The script can be run from vCenter Server 8.x. If there is an issue copying the script to vCenter it may be necessary to change the shell to bash on vCenter.
  Set the default shell on the vCenter Server to bash via the following command to work around this: chsh -s /bin/bash root
• A log named carr.log is created in the folder where the start.sh script is located. For any issues requiring support, please collect this log separately as it will not be collected as part of the support bundle.
• Expired/expiring certificates which are not in use will not be processed by the script. These can be manually deleted in the NSX UI.
• With regards to expiring certificates, only certificates expiring within the next 31 days will be processed, by default, version 1.11 of the script allows you to specify a lead time between 31 and 365 days, using the '-t' options e.g. ./start -t 100 (to check for certificates expiring in the next 100 days). 
• The script processes self-signed certificates only. CA signed certificates are out of scope and must be managed by the organization owners.
• By default the script will run in offline mode, if the appliance has internet connection, you can use the -o option (version 1.11 onwards) to force the script to check online for dependancies.
  
  

Execution Steps:

1. Copy carr-1.11.tar.gz to the client server where it will be run. On the NSX Manager use the /root folder
2. Extract the bundle
   tar -xvf carr-1.11.tar.gz
3. Change to the extracted folder
   cd carr-1.11
4. Launch the carr script
   ./start.sh

Script options since version 1.11:
-o = force online mode
-t = specify lead time for expiring certificates, between 31 and 365 days.

Uninstall:

CARR script gets installed in the directory ~/.virtualenvs/carr_script.
For example, when running CARR script on an NSX Manager, the install can be reversed as follows

     #rm -rf /root/.virtualenvs/carr_script

Note: This rm command deletes files recursively without checks. If executed incorrectly it can remove system files irreversibly requiring the NSX appliance to be replaced.