The carr script only updates a portion of the target APH

search cancel

The carr script only updates a portion of the target APH_TN certificates

book

Article ID: 445807

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

The following output can be observed when executing the carr script:

╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║                                                                              CARR Script Recovery Error Report                                                                               ║
╠════════════════════╦═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣
║ Certificate Checks ║ Error Messages                                                                                                                                                          ║
╠════════════════════╬═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╣
║ APH_TN             ║ There are 10 clients did not join after the APH_TN cert rotation on node: <NSX_Manager_#1_IP>                                                                           ║
║                    ║ bailing out for node: Id: <NSX_Manager_#2_Node_UUID> IP: <NSX_Manager_#2_IP> aph_id: <NSX_Manager_#2_Aph_UUID> is_local: False as bail out is set to true               ║
║                    ║ bailing out for node: Id: <NSX_Manager_#3_Node_UUID> IP: <NSX_Manager_#3_IP> aph_id: <NSX_Manager_#3_Aph_UUID> is_local: True as bail out is set to true                ║
║                    ║                                                                                                                                                                         ║
╚════════════════════╩═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╝

The following output can be observed on carr.log.

$ grep aph_tn_cert_rotation_task carr.log
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:81 - Checking Connection for cert: APH_TN from <NSX_Manager_#1_IP> to other nodes: [<carr.model.cluster_config.Node object at <Memory_Address>>, <carr.model.cluster_config.Node object at <Memory_Address>>]
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:82 - certificate of type: APH_TN replaced on node: Id: <NSX_Manager_#1_Node_UUID> IP: <NSX_Manager_#1_IP> aph_id: <NSX_Manager_#1_Aph_UUID> is_local: False - sleeping for 75 seconds
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:88 - Checking Connection for cert: APH_TN from <NSX_Manager_#1_IP> to other nodes
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:96 - Checking connection for cert: APH_TN from <NSX_Manager_#1_IP> to <NSX_Manager_#2_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:100 - Connection for cert: APH_TN connected from <NSX_Manager_#1_IP> to <NSX_Manager_#2_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:96 - Checking connection for cert: APH_TN from <NSX_Manager_#1_IP> to <NSX_Manager_#3_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:100 - Connection for cert: APH_TN connected from <NSX_Manager_#1_IP> to <NSX_Manager_#3_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_cert_rotation_task.py:125 - For cert: APH_TN, all other nodes connected to <NSX_Manager_#1_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:73 - Checking if  15 TNs have re-connected or not .. sleeping for 75 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:88 - There are still 10 have not joined .. sleeping for 15 secs
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - ERROR - aph_tn_cert_rotation_task.py:102 - There are 10 clients did not join after the APH_TN cert rotation on node: <NSX_Manager_#1_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - ERROR - aph_cert_rotation_task.py:132 - There are 10 clients did not join after the APH_TN cert rotation on node: <NSX_Manager_#1_IP>
<Timestamp> - carr.recovery.aph_tn_cert_rotation_task - MainThread - INFO - aph_tn_cert_rotation_task.py:112 - Restarting : messaging-manager on node: <NSX_Manager_#1_IP>
<Timestamp> - carr.recovery.base_recovery_task - MainThread - ERROR - base_recovery_task.py:102 - Error in recover task: <class 'carr.recovery.aph_tn_cert_rotation_task.AphTNCertRotationTask'> cert_name: APH_TN : error: There are 10 clients did not join after the APH_TN cert rotation on node: <NSX_Manager_#1_IP>
<Timestamp> - carr.recovery.base_recovery_task - MainThread - ERROR - base_recovery_task.py:102 - Error in recover task: <class 'carr.recovery.aph_tn_cert_rotation_task.AphTNCertRotationTask'> cert_name: APH_TN : error: bailing out for node: Id: <NSX_Manager_#2_Node_UUID> IP: <NSX_Manager_#2_IP> aph_id: <NSX_Manager_#2_Aph_UUID> is_local: False as bail out is set to true
<Timestamp> - carr.recovery.base_recovery_task - MainThread - ERROR - base_recovery_task.py:102 - Error in recover task: <class 'carr.recovery.aph_tn_cert_rotation_task.AphTNCertRotationTask'> cert_name: APH_TN : error: bailing out for node: Id: <NSX_Manager_#3_Node_UUID> IP: <NSX_Manager_#3_IP> aph_id: <NSX_Manager_#3_Aph_UUID> is_local: True as bail out is set to true

When executing a dry run of the carr script, it can be observed that only a portion of the APH_TN certificates have been updated:

╔══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╗
║                                                                    CARR Script Validation Report                                                                     ║
╠═════════════════════╦══════════════════════════════════════════════════════════════════════════════════════╦═════════════════════════════════════════════════════════╣
║ Certificate Checks  ║ Validation Results                                                                   ║ Probable Fix                                            ║
╠═════════════════════╬══════════════════════════════════════════════════════════════════════════════════════╬═════════════════════════════════════════════════════════╣
║ APH_TN              ║ SUCCESS: <NSX_Manager_#1_IP>                                                         ║ Certificate of service type: 'APH_TN' will be replaced. ║
║                     ║ WARNING  : <NSX_Manager_#2_IP> : Certificate is expiring in 200                      ║ Certificate of service type: 'APH_TN' will be replaced. ║
║                     ║ days                                                                                 ║                                                         ║
║                     ║ WARNING  : <NSX_Manager_#3_IP> : Certificate is expiring in 200                      ║                                                         ║
║                     ║ days                                                                                 ║                                                         ║
║                     ║                                                                                      ║                                                         ║
╚═════════════════════╩══════════════════════════════════════════════════════════════════════════════════════╩═════════════════════════════════════════════════════════╝

Environment

VMware NSX

Cause

After the APH_TN certificate is updated, connectivity between the NSX Manager and the Transport Node is checked. If the connection cannot be verified within a certain period of time, the process times out, and subsequent certificate updates are aborted.

Resolution

Resolve the connectivity issue between the NSX Manager and the Transport Node, and then execute the carr script again. To resolve the connection issue, as also stated in KB#369034, first follow the resolution section on KB#417130.

Additional Information

KB#369034 : Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX

KB#417130 : Host Disconnected after certificate expiration and replacement, Transport Nodes connection status is timing out at 48%. ("Waiting for Connection to Managers")

Feedback

thumb_up Yes

thumb_down No