Edge VM MPA connectivity is down after certificate replacement
search cancel

Edge VM MPA connectivity is down after certificate replacement

book

Article ID: 389595

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  1. After replacing the Certificate on the manager we will see an "MPA Connect" error on the Edge similar to the following 

 

 

  1. On the manager node, you will see entries similar to the following in the /var/log/proton/nsxapi.log file:

 

2025-02-27T22:24:33.482Z  INFO UfoIndexer-BatchExecutor-search_manager-2 EdgeTNValidationUtils 5296 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Set FN state error MPA disconnected TRANSPORT_NODE_SYNC_PENDING

 

2025-02-27T22:24:33.482Z  INFO UfoIndexer-BatchExecutor-search_manager-2 EdgeTNValidationUtils 5296 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [entId=/infra/sites/default/enforcement-points/default/edge-transport-node/0000-0000-0000-00] Edge either in error state, not ready or mpa disconnected, failure code: 0,state:MPA_DISCONNECTED, mpa_connection: false

 

-----------------------------------------------

 

2025-02-27T22:20:02.622Z ERROR WrapperStartStopAppMain TrustStoreServiceImpl 1700520 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] Failed to sync certificate between DB and disk for profile: profileName: Message Bus Client for K8S Platform, serviceType: K8S_MSG_CLIENT, preProcessor: com.vmware.nsx.management.cloudnative.pre_processor.KafkaMsgClientCertPreProcessor, postProcessor: null, uniqueUse: false, clusterCertificate: true, requiresPrivateKey: true, nodeTypes: [global-manager, nsx-manager, nsx-shared], alias: k8s-msg-client, keyStorePath: /home/secureall/secureall/.store/.bluelane_keystore, keyStorePasswordPath: /config/http/.http_cert_pw

 

 

  1. On the manager node, the files under /etc/vmware/nsx-appl-proxy/ have permissions similar to the following (ls -la /etc/vmware/nsx-appl-proxy/)

 

-rw-r-----  1 uproton    uproton    1.7K Feb 28 16:22 appl-proxy-privkey.pem

-rw-r-----  1 uproton    uproton    1.7K Feb 27 22:20 appl-proxy-privkey.pem.

-rw-r-----  1 uproton    uproton    1.7K Feb 27 22:15 appl-proxy-privkey.pem.

-rw-rw-r--  1 appl-proxy appl-proxy 1.3K Feb 27 22:15 appl-proxy-ar-cert.pem

-rw-r-----  1 uproton    uproton    1.3K Feb 27 22:15 appl-proxy-ar-cert.pem.

-rw-rw-r--  1 appl-proxy appl-proxy 1.7K Feb 27 22:15 appl-proxy-ar-privkey.pem

-rw-r-----  1 uproton    uproton    1.7K Feb 27 22:15 appl-proxy-ar-privkey.pem

 

 

  1. On the faulty edge, messages similar to the following are seen in the /var/log/syslog file:

 

2025-02-28T16:47:49.023Z  NSX 2007 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-net" tid="3702" level="INFO"] StreamSocket[754 Open f:64 i:199414 ? -> ssl://#.#.#.#:1234] on_connect 167772294-certificate verify failed (SSL routines)

2025-02-28T16:47:49.024Zedge name NSX 2007 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-net" tid="3702" level="WARNING"] StreamConnection[754 Connecting to ssl://#.#.#.#:1234 sid:754] Couldn't connect to 'ssl://<ip_of_the_manager> (error: 167772294-certificate verify failed (SSL routines))

2025-02-28T16:47:49.024Z NSX 2007 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-net" tid="3702" level="WARNING"] StreamConnection[754 Error to ssl://#.#.#.#:1234 sid:-1] Error 167772294-certificate verify failed (SSL routines)

2025-02-28T16:47:49.024Z NSX 2007 - [nsx@6876 comp="nsx-edge" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="3702" level="WARNING"] RpcConnection[754 Connecting to ssl://#.#.#.#:1234 0] Couldn't connect to ssl://#.#.#.#:1234 (error: 167772294-certificate verify failed (SSL routines))

 

Cause

Changing the certificates on the manager nodes may cause some discrepancy on the on the manager certificate thumbprint.

Resolution

Workaround

  1.  Get the certificate thumbprint from each manager node. 
    Manager> get certificate api thumbprint
  2. SSH to the faulty edge node as the admin user.
  3. Run the following commands:
    push host-certificate <manager-IP-FQDN> username <username> thumbprint <cert-api-thumbprint-of-manager> password <password>

    sync-aph-certificates <manager-IP-FQDN> username <username> thumbprint <cert-api-thumbprint-of-manager> password <password>
  4. Repeat Step 3 for each manager node thumbprint.
  5. Switch to root (st en, enter root password when prompted)
  6. Run the following command
    /etc/init.d/nsx-proxy restart

Additional Information

Powered by Wolken Software