With IDS/IPS configuration causes corfu compactor in to Out Of Memory issue in 3.1.2/3.1.2.1

Products

• VMware vDefend Firewall

Issue/Introduction

• NSX-T Cluster status is unavailable and cannot access the UI
• Core files created in /image/core/ directory

Environment

VMware NSX-T Data Center 3.x

Cause

Caused Corfu nonconfig tables (IDS/IDP, system tables), Corfu compresses data before writing in to the DB. This compression ratio depending on the table’s contents. In the cases where the compression ratio is very high (usually greater than compactor JVM size), will see OutOfMemory error.

/var/log/corfu-nonconfig

zgrep 'Core dump written' nonconfig-corfu-compactor-audit.*

nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.2403
nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.23613
nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.12216
nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.31865
nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.21275
nonconfig-corfu-compactor-audit.1.log.gz:# Core dump written. Default location: //core or core.9438

nonconfig-corfu-compactor-audit.log file
2023-11-14:20:19.234Z INFO main UfoCompactor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfu-compactor"] Completed checkpoint namespace: security_data_service, tableName: url_reputations, with 6 entries in 819 ms
2023-11-14:20:19.456Z INFO main UfoCompactor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfu-compactor"] Opening table ids_event_data in namespace security_data_service. Disk-backed: true
2023-11-14:20:19.567Z INFO main SMRObject - ObjectBuilder: open Corfu stream security_data_service$ids_event_data id <UUID>
2023-11-14:20:19.678Z INFO main UfoCompactor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfu-compactor"] Starting checkpoint namespace: security_data_service, tableName: ids_event_data
2023-11-14:20:19.789Z INFO main MultiCheckpointWriter - appendCheckpoints: appending checkpoints for 1 maps
2023-11-14:20:19.123Z INFO main CheckpointWriter - appendCheckpoint: Started checkpoint for <UUID> at snapshot Token(epoch=38, sequence=15376194)
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /image/core/java_pid29495.hprof ...
Dump file is incomplete: No space left on device

Resolution

This issue is resolved in VMware NSX-T Data Center 3.1.3 and 3.2.0