SCS Config Server service instance fails to synchronize mirrors with the error, "Load key /var/vcap/store/mirror/####/private-key: error in libcrypto"

Products

• VMware Tanzu Application Service

Issue/Introduction

It is found that the mirror service fails to sync with remote git repository using ssh after pressing the button, "SYNCHRONIZE MIRRORS".

 

When reviewing the log files, you might notice the message below, "error in libcrypto".

2024-##-##T##:##:##.##Z  INFO 6 --- [ctor-http-nio-2] i.p.s.c.mirrorservice.mirrors.ShellUtil  : Load key "/var/vcap/store/mirror/####/private-key": error in libcrypto 2024-##-##T##:##:##.##Z  INFO 6 --- [ctor-http-nio-2] i.p.s.c.mirrorservice.mirrors.ShellUtil  : ###@example.com: Permission denied (publickey). 2024-##-##T##:##:##.##Z  INFO 6 --- [ctor-http-nio-2] i.p.s.c.mirrorservice.mirrors.ShellUtil  : fatal: Could not read from remote repository.

Cause

The "error in libcrypto" message typically indicates an issue with the OpenSSL library, which is used for cryptographic operations.This error can occur for various reasons, such as:

1. Incorrect or corrupted key files: If the private or public key files are corrupted or not in the expected format, OpenSSL may fail to load them, resulting in this error.

2. Passphrase issues: If a passphrase is required for the key and it is incorrect or not provided, OpenSSL may report this error.

3. Library version mismatch: Sometimes, upgrading or downgrading OpenSSL can cause compatibility issues with existing keys or applications.
4. Permissions issues: Incorrect file permissions on the key files can prevent OpenSSL from accessing them properly.

Resolution

We can narrow down this issue by accessing the remote repository with the current key file. Here is an example,

/var/vcap/data/packages/git/####/bin/git -c "core.sshCommand=ssh -i /tmp/your_current_private_key -o StrictHostKeyChecking=no" clone ssh://####@example.com:####/your_repository

 

If it returned error, it is recommended to generate a new key file with the following command as an example to fix this issue.

ssh-keygen -t ed25519 -C "[email protected]"

 

Once the new key is ready, the config-server service instance can be refreshed with the following command,

cf update-service config-server -c '{"git": { "uri": "[email protected]/spring-cloud-services-samples/cook.git", "hostKey": "EXAMPLEcccc1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+...", "hostKeyAlgorithm": "ssh-rsa", "privateKey": "-----BEGIN EXAMPLE RSA PRIVATE KEY-----\nMIIJKQIB..."} }'