Troubleshooting and Understanding Hardware Security Module (HSM) Implementation on the BOSH Director

Products

VMware Tanzu Platform Core

Issue/Introduction

This article outlines the deployment architecture of SafeNet Luna Hardware Security Module (HSM) integration with CredHub on the BOSH Director, highlights key architectural differences between Director-level and Tanzu Application Service (TAS) implementations, and provides step-by-step troubleshooting procedures when the Director fails to boot due to HSM configuration issues.

Environment

Products: Tanzu Operations Manager, BOSH Director, CredHub
Encryption Provider: SafeNet Luna Network HSM
Components: credhub-release, luna-hsm-client

Cause

Hardware Security Module (HSM) deployment failures and subsequent BOSH Director startup timeouts typically stem from one of the following root causes:

Java & Native Client Version Mismatch: Modern CredHub packages bundled with Java 21+ or Java 25 fail to load or initialize legacy Luna HSM Client native libraries (such as Luna v7.4.0), leading to process hangs or library loading errors during startup.
Configuration Script Syntax Errors: Unescaped special characters (e.g., $ signs) in partition passwords cause bash interpolation failures during the execution of configure_hsm.erb / configure_hsm.sh.
Credential Scope Mismatch: The partition_password specified in Ops Manager does not match the Crypto Officer (CO) role password set on the SafeNet Luna HSM partition.
Network & Initialization Deadlocks: Network connectivity issues between the Director VM and the HSM appliance prevent the HA group or handshake from completing, causing the Spring Boot startup process to stall indefinitely.

Resolution

Troubleshooting a failing BOSH Director HSM deployment

Step 1: Recover Logs Out-of-Band

Because bosh ssh is unavailable when the BOSH Director VM cannot complete its boot sequence:

Power down the failing BOSH Director VM.
Attach the Director’s persistent/data disk to a temporary Rescue VM running in the same infrastructure.
1. (See article: How to detach disk from unresponsive vm to collect logs)
Mount the disk (for example, to /mnt/director_disk).

Step 2: Inspect Job Logs

Navigate to the log directory on the mounted disk:

cd /mnt/director_disk/var/vcap/sys/log/credhub/

Check the log files in the following sequence to diagnose the issue:

configure_hsm.stderr.log: Check for shell syntax or variable errors. Look specifically for bash interpolation failures (e.g., unbound variable errors caused by unescaped $ characters in partition passwords) or missing directory paths (such as legacy /lib/ext/ paths under Java 21+).
configure_hsm.stdout.log: Verify whether the HA setup script completed successfully. Look for log lines indicating Creating new HSM HA group or HSM HA appears to be initialized. Skipping HSM HA setup.
credhub.stderr.log & credhub.stdout.log: If configure_hsm exited cleanly with code 0, inspect these log files for Java application startup errors. Log entries showing Successfully connected to UAA followed by an indefinite pause indicate the Spring Boot process started, but is stalled waiting on native crypto library initialization or an underlying HSM network handshake.

Step 3: Replicate Java Startup Manually (Optional)

To capture real-time Java stack traces without booting the full Director VM, chroot into the mounted Director disk on your Rescue VM:

Bind required filesystems:

sudo mount --bind /dev /mnt/director_disk/dev
sudo mount --bind /proc /mnt/director_disk/proc
sudo mount --bind /sys /mnt/director_disk/sys

Chroot into the Director environment:
```
sudo chroot /mnt/director_disk
```

Source environment variables and launch CredHub manually:

source /var/vcap/jobs/credhub/tmp/var-store
export JAVA_HOME=/var/vcap/packages/openjdk_25.0/jre
export JAVA_TOOL_OPTIONS="-Xmx1024m -Dlog4j.configurationFile=/var/vcap/jobs/credhub/config/log4j2.properties -Djava.security.egd=file:/dev/urandom -Djava.io.tmpdir=/var/vcap/data/credhub/exec-tmp"
cd /var/vcap/packages/credhub
$JAVA_HOME/bin/java -cp credhub.jar org.springframework.boot.loader.launch.JarLauncher --management.server.port=8845

Step 4: Remediation and Fix Steps

Based on the root cause identified in Steps 2 and 3, apply the corresponding fix:

Escape Password Characters: If shell interpolation errors were logged in configure_hsm.stderr.log, update Ops Manager to escape special characters (e.g., replace $ with \$) or update the partition password to use alphanumeric characters only.
Verify Password Scope: Access the SafeNet Luna HSM appliance directly via SSH (lunacm) and ensure the password configured in Ops Manager matches the Crypto Officer (CO) role password for the partition, rather than the global Security Officer (SO) password.
Resolve Client / JDK Incompatibility: If native library initialization fails under JDK 21+ or JDK 25, upgrade the luna-hsm-client package in your stemcell/release to a version compatible with modern Java runtimes instead of legacy v7.4.0.
Verify Network Connectivity: Ensure port 1792 is open and unblocked between the BOSH Director subnet and the HSM appliances.
Unmount and Re-deploy: Exit the chroot environment, unmount all bind mounts (/dev, /proc, /sys), detach the persistent disk from the Rescue VM, reattach it to the BOSH Director VM, and apply changes in Ops Manager to complete the deployment.

Additional Information

Feature	BOSH Director CredHub	TAS / App Runtime CredHub
Placement	Co-located as an embedded job directly on the BOSH Director VM.	Deployed as independent application instances (`credhub/0`, `credhub/1`) on dedicated worker VMs.
Configuration	Configured via Ops Manager > BOSH Director Tile > Director Config /CredHub Encryption Provider	Configured via Elastic Application Runtime Tile > CredHub.
Impact of Failure	Control Plane Outage: Prevents the BOSH Director VM from completing deployment or booting.	Runtime Issue: The BOSH Director stays online; only app/runtime secret operations fail.
Accessibility on Failure	Out-of-Band Required: `bosh ssh` is unavailable because the Director/daemon is unreadable or down.	Standard Access: `bosh ssh credhub/0` remains fully functional.

Key Takeaways

Shared release dependencies: Both BOSH Director CredHub and TAS CredHub utilize the same underlying credhub-release BOSH package, configuration templates (configure_hsm.erb / configure_hsm.sh), and bundled Luna HSM client libraries.
Java / client library compatibility: Modern versions of CredHub ship with modern Java runtimes (Java 21+ / Java 25). Legacy Luna HSM Client versions (such as v7.4.0) are only supported on legacy Java versions (up to Java 11). Running an outdated Luna client library on a newer JDK can lead to process stalls or native library loading failures.
Password scope: The partition_password (Crypto Officer / CO role) defined in the CredHub configuration is distinct from the global HSM Administrator / Security Officer (SO) password set via hsm init. The password configured in Ops Manager must match the Crypto Officer partition password on the HSM appliance.