Error: CARR Script Fails With 503 Service Unavailable or 409 Conflict Due to NSX Management Plane Overload
search cancel

Error: CARR Script Fails With 503 Service Unavailable or 409 Conflict Due to NSX Management Plane Overload

book

Article ID: 451547

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • When executing the Certificate Analyzer, Results and Recovery (CARR) script on an NSX Manager, the script terminates prematurely or fails to update specific nodes.

  • The CARR script displays the following error:

    Error in checking TNs connected back status: 503 Server Error: Service Unavailable for url

  • The carr.log file indicates a 503 Service Unavailable or 409 Conflict error, often accompanied by messages regarding the server being overloaded: 

    503 Server Error: Service Unavailable for url: https://####:443/api/v1/trust-management/certificates/action/replace-host-certificate/####

Environment

VMware NSX

Cause

The NSX Manager is overloaded due to excessive API calls. We recommend investigating the source of high API request volume. Refer to: Troubleshooting NSX API Calls.

Resolution

To mitigate the management plane load and allow the operation to complete, the CARR script operations can be batched:

  1. Run Dry Run: Ensure you have executed the script with the -d flag to generate the dry_run_transport_nodes_validation_report.yaml.

  2. Modify Configuration for Batching: Open the validation_config_recovery_mode.yaml file generated by the dry run.

  3. Restrict Processing Scope: Identify the nodes listed in the configuration file. To prevent overloading the management plane, manually edit the file to set the apply_fix status to false for the majority of nodes. Select a small, incremental subset (e.g., 5-10 nodes) and set their apply_fix status to true.

  4. Execute in Increments: Run the CARR script to apply the changes to the restricted subset.

  5. Monitor Management Plane: After the script finishes the batch, verify the successful update in the NSX UI (System > Certificates) and allow the management plane a short recovery period to process pending tasks.

  6. Iterate: Repeat the process with the next small batch of nodes until the entire environment is remediated.

Additional Information

Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX

Powered by Wolken Software