SSP Status Degraded, Intelligence/NDR intermittently Down, and Policy Recommendations show "Nothing to Recommend" due to MinIO storage quorum loss

Products

• VMware vDefend Firewall

Issue/Introduction

• SSP Platform Health: Status shows as Degraded in the SSP UI.
• Feature Availability: Intelligence and NDR/NTA features show as Down intermittently.
• Policy Recommendations: Running a recommendation returns "Nothing to Recommend" despite active flows.
• Alarms: "SSP Alarm - Service Down - Platform Service pcap-storer-pcapstorer is degraded."

Environment

SSP 5.1.1.1 and earlier versions

Cause

MinIO services may experience "Insufficient Read/Write Quorum" errors, leading to degraded performance or service unavailability within the platform. This behavior typically manifests when the underlying storage filesystem enters a read-only state, preventing the platform from persisting new data or reading existing configuration and flow information.

The underlying ext4 journal daemon (jbd2) on multiple worker nodes hung, becoming blocked while waiting for write operations to complete. This state persisted for an extended duration (escalating from 120 seconds to over 240 seconds). Because the host storage (vSAN datastore) failed to respond to write requests, the guest operating system's filesystem integrity mechanisms were triggered. To prevent data corruption following the prolonged I/O stall, the ext4 filesystems performed an emergency journal abort and forced a remount into a read-only state.

Because standard Kubernetes liveness and readiness probes do not perform disk-write operations, the affected pods may continue to pass health checks, leaving the platform in a "Ready" but non-functional state.

Resolution

PVCs in a locked read-only state cannot recover automatically. You must verify the state and manually restart statefulsets to force volume remounts, then reinitialize any Druid supervisors that remain in an UNHEALTHY_TASKS state.

1. Immediate Log Collection

Collect all support bundles (VC, vSAN, NSX, SSPI, SSP) immediately. Kernel logs roll over quickly; capturing them early is critical to confirm the transient storage I/O stall.

2. Verify Root Cause (CLI Verification)

 Check the SSPI support bundle for kernel-level logs in /system/journalctl_--utc_-b_all_-k indicating filesystem errors. Specifically, look for jbd2 (journal daemon) blocks or EXT4-fs errors:

• • Example Log Entry: EXT4-fs (sdc): Remounting filesystem read-only

  • If you do not see these specific entries, the issue may be due to a different component failure.

Attempt to create a file inside a Minio pod to confirm the read-only state:
SSH To SSP Installer as sysadmin;

k exec -it minio-2 -n nsxi-platform -- touch /data/minio/testfile

# Expected failure if root cause is correct:
# touch: cannot touch 'testfile': Input/output error

3. Restart StatefulSets

Perform a rolling restart of the affected statefulsets to clear the read-only mount status.

k rollout restart statefulset/minio -n nsxi-platform

Verification: After the restart, the touch command in Step 2 should succeed.

4. Monitor nsx-config Readiness

The nsx-config pods may stay in a 0/2 Running  state if they are blocked waiting for Druid. Check the logs for supervisor health:

k logs <nsx-config-pod-name> -c wait-for-druid-supervisor-ready -n nsxi-platform

Look for: "Supervisor: pace2druid_manager_realization_config status: UNHEALTHY_TASKS"

5. Reinitialize Druid Supervisors

If nsx-config is not ready or recommendations show "Nothing to Recommend," reset the unhealthy Druid supervisors:

1. Exec into the Druid Router pod

   k exec -it <druid-router-pod-name> -n nsxi-platform -- /bin/bash

2. Identify Unhealthy Supervisors
   
    

   curl https://localhost:8280/druid/indexer/v1/supervisor?state=true -k
   
   
3. Reset the Unhealthy Supervisor
   
    

   curl --request POST "https://localhost:8280/druid/indexer/v1/supervisor/<supervisor-id>/reset" -k

6. Final Verification

Once the steps above are completed, perform the following checks in the SSP UI:

• Platform Health: Navigate to the System tab and verify the SSP Status is Stable.
• Feature Status: Confirm both Intelligence and NDR features show as UP without intermittent flapping.
• Recommendations: Run a new Policy Recommendation for a target VM. The engine should now successfully process flows and provide recommendations instead of "Nothing to Recommend."
• Alarms: Verify that the pcap-storer-pcapstorer and other SSP-related alarms have resolved and are no longer firing.