"Client handshake failed for peer 2: certificate verify failed" error when starting the operator in VMware Blockchain

Products

• VMware Blockchain

Issue/Introduction

Symptoms:

• After a maintenance operation, such as scaling up or scaling down the blockchain, the operator fail to start.
• In the operator logs on the VMware Blockchain client node running the operator, you see an error similar to:

2022-12-08T07:42:04,169Z|ERROR||concord-bft.tls.conn||||AsyncTlsConnection.cpp:470|std::pair<bool, NodeNum> bft::communication::tls::AsyncTlsConnection::checkCertificate(X509 *, std::string, const std::string &, std::optional<NodeNum>)|X509_cmp failed at node: 105, type: server, peer: 2 res=15 2022-12-08T07:42:04,169Z|ERROR||concord-bft.tls.connMgr||||TlsConnectionManager.cpp:228|void bft::communication::tls::ConnectionManager::onClientHandshakeComplete(const asio::error_code &, bft::communication::NodeNum)|Client handshake failed for peer 2: certificate verify failed 2022-12-08T07:42:04,173Z|WARN ||concord-bft.tls.conn||||AsyncTlsConnection.cpp:192|void bft::communication::tls::AsyncTlsConnection::dispose(bool)|Closing connection to node 2

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs when the encrypted private key is no longer valid and needs to be refreshed.

Resolution

To resolve this issue, recreate the encrypted key for the operator.

1. SSH to the operator node.
2. Identify the blockchain private key.
3. Generate a public key from the private key:

Note: One location you might find the private key is /config/daml-ledger-api/concord-operator
cd /config/daml-ledger-api/concord-operator
openssl pkey -in <your-private-key.pem> -out public-key.pem -outform PEM -pubout

4. Compare the public key generated against the one in the json file used for the deployment/scaling reconfigure.

If they match, on the operator node proceed with the following steps:

1. Delete the operator:

docker rm -f operator

2. Delete the old encrypted key:

cd /config/daml-ledger-api/concord-operator
mv operator_priv.pem /tmp
mv operator_priv.pem.enc /tmp

3. Generate the new encrypted key:
   1. If you are running version 1.5 or earlier, run:

image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image

2. If you are running version 1.6 or later, run:

image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image

4. When prompted to paste private operator key and press Ctrl+d.

Note: After this a new /config/daml-ledger-api/concord-operator/operator_priv.pem.enc file will be generated.

5. Start the operator:
   1. If you are running version 1.5 or earlier, run:

image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator");docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image

2. If you are running version 1.6 or later, run:

image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image

6. Check operator file to verify the issue is resolved:

docker logs operator | less