vDefend SSP Container Volume Provisioning Fails with Database Temporarily Unavailable Error

Products

• VMware vSphere ESXi

Issue/Introduction

When deploying vDefend Security Services Platform (SSP) components in a VMware vSphere environment with HPE Alletra MP storage, the following symptoms occur:

• Kubernetes Persistent Volume Claims (PVCs) remain in a Pending state.

• The vSphere Client "Recent Tasks" view shows "Create a virtual disk object" failing with the status: "Database temporarily unavailable or has network problems".
  

• vCenter vpxd.log reports: vim.vslm.vcenter.VStorageObjectManager.createDisk: :vim.fault.DatabaseError --> Result: (vim.fault.DatabaseError) { msg = "Database temporarily unavailable or has network problems." }

• ESXi hostd.log shows repeated failed attempts to obtain a lock: Unable to obtain a tick for: kAttempts = 32.

• ESXi vmkernel.log reports ATS miscompares: ScsiDeviceIO: <REDACTED_PII>: Cmd(0x45ea12d83100) 0x89, CmdSN 0x1b3610 from world <REDACTED_PII> to dev "naa.<REDACTED_SECRETS>" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0

• ESXi vmkernel.log reports lock errors: DLX: <REDACTED_PII>: vol '<REDACTED_HOSTNAMES>', lock at 349773824: Lock type: 10C00001. [Req mode 1] try lock error: Atomic test and set of disk block returned false for equality

• VOMA Dump reports: ON-DISK ERROR: No hash block entry for directory entry (1, 0)

Environment

ESXi 8.0 U3
SSP 5.1
SSP 5.11

Cause

The issue is caused by VMFS inode inconsistency within the catalog directory of the HPE Alletra backed datastore. The directory entry hash metadata links associated with the vclock-<REDACTED_SECRETS> file are corrupted or missing. Because the metadata links are broken, ESXi hosts fail to acquire ATS locks for the file, leading to constant liveness checks and tick failures. After 32 retries, the Virtual Storage Lifecycle Manager (VSLM) times out, returning a generic DatabaseError to vCenter.

Resolution

Workaround the problematic datastore folder:

• Shut down the SSP and SSPI virtual machines to halt automated provisioning.

• Unmount the affected datastore from all ESXi hosts in the cluster.

• Provision a new datastore.

• Redeploy the SSP architecture onto the newly provisioned datastore.

 

 

Additional Information

ATS miscompare events
PVC creation is stuck in pending state