"Failed to verify LDAP server with error: error: ldap server network timeout" error after upgrading Harbor to v2.13.2

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

After upgrading the Harbor tile to v2.13.2, LDAP authentication fails.  

When using the "Test LDAP server" button from Harbor configuration authentication page, the following error pops up.

"Failed to verify LDAP server with error: error: ldap server network timeout"

Environment

VMware Tanzu Kubernetes Grid Integrated Edition

Harbor Registry v2.13.2

Cause

This issue is due to the post-quantum (PQ) cryptographic schemes being used by default in Go 1.23+ (used in Harbor v2.13.2), which makes the Client Hello being sent in 2 TCP frames (golang issue).  Such kind of schemes are generally large and increase the packet size significantly.  This causes issues in firewalls or old servers that do not yet support such algorithms.

If the AD/LDAP servers are fronted by a Load Balancer, the issue could be experienced intermittently if there is a mix of new and old versions of the AD/LDAP servers, as the newer versions likely support the PQ cryptographic schemes and TLS connection would be successful in that case.

Resolution

The ideal resolution to this is to upgrade the AD/LDAP server to a newer version that supports TLS v1.3 and Post-Quantim Cryptographic schemes.

As a workaround in Harbor, you can disable the PQ schemes by following the steps below.  The workaround leverages the BOSH os-conf release to make the change persistent through VM recreation and deployment upgrades.

Steps:

1. Upload the os-conf release to BOSH Director

bosh upload-release --sha1=sha256:efcf30754ce4c5f308aedab3329d8d679f5967b2a4c3c453204c7cb10c7c5ed9 "https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=23.0.0"

 

2. Create a BOSH runtime config file "osconf-runtime.yml" with the following contents.  Replace the Harbor deployment name accordingly.

releases:
- name: os-conf
  version: 23.0.0

addons:
- name: osconf-prestart-config-for-harbor
  jobs:
  - name: pre-start-script
    release: os-conf
    properties:
      script: |-
         #!/bin/bash
         sleep 60  # wait for 1 minute to allow Harbor pre-start to complete first.
         while :
         do
           time=`date`
           harborprestartstatus=`ps -ef | grep /var/vcap/jobs/harbor/bin/pre-start | grep -v grep`
           if [ -z $harborprestartstatus ]
           then
             echo "$time - Harbor pre-start is completed."
             break
           else
             echo "$time - Harbor pre-start is still running.  Sleeping 5 secs."
             sleep 5
           fi
         done
         time=`date`
         if [ -f /var/vcap/packages/harbor-app/common/config/core/env ]; then
             echo "GODEBUG=tlsmlkem=0" >> /var/vcap/packages/harbor-app/common/config/core/env
             echo "$time - GODEBUG=tlsmlkem=0 added to /var/vcap/packages/harbor-app/common/config/core/env"
             cat /var/vcap/packages/harbor-app/common/config/core/env
         else
             echo "$time - /var/vcap/packages/harbor-app/common/config/core/env does not exist.  Skipping."
         fi
include:
  deployments: [ harbor-container-registry-xxx ]

 

3. Run the following command to install the runtime config:

bosh update-runtime-config --name=osconf-prestart-config-for-harbor osconf-runtime.yml

 

4. Run "bosh configs" to see the list of the active runtime configs.  You should then see one with a name of "osconf-prestart-config-for-harbor".

5. Save the manifest of the Harbor deployment into a file, and then deploy the manifest.

bosh -d harbor-container-registry-xxx manifest > harbor-manifest.yml
bosh -d harbor-container-registry-xxx deploy harbor-manifest.yml

 

6.  After the deployment, verify that the "GODEBUG=tlsmlkem=0" was appended.

bosh -d harbor-container-registry-xxx ssh harbor-app -c "sudo cat /var/vcap/packages/harbor-app/common/config/core/env"