(SSP) Security Intelligence - Recommendations Stuck in "Queued for Discovery" due to vSphere CSI Storage Policy Mismatch

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

When attempting to run system or security recommendations within the platform interface, the task fails to initiate and becomes stuck indefinitely. This is caused by an underlying storage provisioning failure where the vSphere Container Storage Interface (CSI) cannot create the required persistent disks due to a named storage policy mismatch between Kubernetes and vCenter.

Symptoms

Users and administrators may observe the following behavior across the application and infrastructure layers:

Platform UI Symptoms

Security or system recommendations are stuck in a "Queued for Discovery" state.
Users are unable to execute or advance the recommendation process.
A warning or alert banner is displayed stating to "add worker nodes", even if physical or virtual compute nodes are already present and running.
Logging into CLI of SSP-Installer and executing k get pods -A | grep -v Completed | grep -v Running shows application or data-processing pods in the platform namespace remain in a Pending or ContainerCreating state.
Running k get pvc -A shows multiple Persistent Volume Claims (PVCs) for core analytical engines (e.g., data correlators, streaming jobs) stuck in a Pending status.
Describing a pending PVC (k describe pvc <pvc-name> -n <namespace>) reveals the following explicit error in the Events section:
```
Warning  ProvisioningFailed  ...  csi.vsphere.vmware.com_vsphere-csi-controller-...  f
```

Environment

vDefend Security Services Platform(SSP) - 5.1.1

Cause

This issue is caused by Configuration Drift between the vSphere infrastructure storage profiles and the Kubernetes StorageClass configurations.

The Kubernetes StorageClass contains a hardcoded parameter string (storagepolicyname) that it uses to request VMDK/FCD allocations from vCenter via VMware's Profile-Based Management (PBM) API. If the storage policy in vCenter is renamed, optimized (e.g., changed from a specific layout like RAID6 to an automated policy like AutoRAID), or modified without updating the corresponding Kubernetes manifests, vCenter will return a NotFound error.

Because the CSI driver cannot validate the storage policy, it refuses to provision the disk, leaving the analytics pods without storage, which ultimately blocks the platform from running recommendations.

Resolution

Step 1: Identify the Correct Storage Policy Name in vCenter

Log into the vSphere Client HTML5 UI.
From the main menu, navigate to Policies and Profiles > VM Storage Policies.
Locate the active storage policy assigned to the datastores hosting your SSP worker nodes.
Note the exact string name of the active policy (e.g., CL-MGMT-WLD01 - Optimal Datastore Default Policy - AutoRAID), paying close attention to spaces, dashes, and capitalization.

Step 2: Backup the Existing StorageClass Configuration

Log into your cluster management console and export the current storage class layout to a backup file:

k get storageclass <storage-class-name> -o yaml > sc_backup.yaml

Step 3: Modify the StorageClass Configuration File

Open the sc_backup.yaml file in a text editor (such as vi or nano):

nano sc_backup.yaml

Locate the parameters block.
Update the storagepolicyname value to match the exact string verified from vCenter in Step 1.
Clean up system-generated metadata fields to avoid conflicts during re-application (delete lines containing resourceVersion, uid, and creationTimestamp).

Example of a corrected parameters block:

parameters:
  storagepolicyname: "LAB-MGMT-LAB01- Default Policy - AutoRAID"
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Step 4: Replace the Mismatched StorageClass

Because StorageClass parameters are immutable in a live cluster, the resource must be deleted and recreated:

# Delete the old configuration
k delete storageclass <storage-class-name>

# Apply the updated configuration
k apply -f sc_backup.yaml

Step 5: Verify the Fix and Recommendation Pipeline

Monitor PVC Status: Since the volume binding mode is typically Immediate, the vSphere CSI controller will immediately pick up the change on its next retry loop. Monitor the PVCs until they transition to Bound:
```
k get pvc -A -w
```
Monitor Pod Status: Once the storage binds, verify that the dependent platform processing pods transition to a Running state:
```
k get pods -A | grep -v Running | grep -v Completed
```
Execute Recommendation: Return to the platform UI. The "add worker nodes" warning will disappear, and you can successfully run the recommendation pipeline.