Intermittent PVC/Storage Errors Causing Pod Failures and Potential SSP Platform Degradation.

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

SSP may experience PVC/storage errors if the underlying host storage layer fails. In this scenario, pod logs may show errors like 'CnsFault error: VSLM task failed', and pods may enter states such as CrashLoopBackOff, ContainerCreating, or Init. This can lead to degraded performance or failure in SSP platform operations. The storage layer failure could also cause issues like "Database temporarily unavailable" in vCenter logs, affecting the ability to attach or manage disks.

Environment

SSP 5.0

Cause

The underlying host used by SSP experiences a power outage or failure and the corresponding host will show the below errors in vpxd log (/var/log/vmware/vpxd) in Vcenter :

 

"retrieveVStorageObjectPathAndCrypto: Database temporarily unavailable or has network problems."

Resolution

Resolve the underlying host issues (due to power failure), and the SSP storage will recover.