NSX metrics consume a large disk space on NSX Manager nodes

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are running NSX-T 3.x
• /var/log/corfu is consuming a large about of space

1.7G /var/log/cbm
2.5G /var/log/cloudnet
8.8G /var/log/corfu
2.5G /var/log/journal
2.0G /var/log/proton
1.8G /var/log/vmware

• Files similar to the below are observed in /var/log/corfu

corfu.infrastructure.message-handler.bootstrap_sequencer.csv
corfu.infrastructure.message-handler.committed_tail_request.csv
corfu.infrastructure.message-handler.compact_request.csv
corfu.infrastructure.message-handler.failure_detector_metrics_request.csv
corfu.infrastructure.message-handler.flush_cache.csv
corfu.infrastructure.message-handler.inspect_addresses_request.csv
corfu.infrastructure.message-handler.keep_alive.csv
corfu.infrastructure.message-handler.known_address_request.csv
corfu.infrastructure.message-handler.layout_bootstrap.csv
corfu.infrastructure.message-handler.layout_committed.csv
corfu.infrastructure.message-handler.layout_prepare.csv
corfu.infrastructure.message-handler.layout_propose.csv
corfu.infrastructure.message-handler.layout_request.csv
corfu.infrastructure.message-handler.log_address_space_request.csv
corfu.infrastructure.message-handler.management_bootstrap_request.csv
corfu.infrastructure.message-handler.management_failure_detected.csv
corfu.infrastructure.message-handler.management_healing_detected.csv
corfu.infrastructure.message-handler.management_layout_request.csv
corfu.infrastructure.message-handler.node_state_request.csv
corfu.infrastructure.message-handler.orchestrator_request.csv
corfu.infrastructure.message-handler.ping.csv
corfu.infrastructure.message-handler.prefix_trim.csv
corfu.infrastructure.message-handler.range_write.csv
corfu.infrastructure.message-handler.read_request.csv
corfu.infrastructure.message-handler.reset.csv
corfu.infrastructure.message-handler.reset_logunit.csv
corfu.infrastructure.message-handler.restart.csv
corfu.infrastructure.message-handler.seal.csv
corfu.infrastructure.message-handler.sequencer_metrics_request.csv
corfu.infrastructure.message-handler.sequencer_trim_req.csv
corfu.infrastructure.message-handler.streams_address_request.csv
corfu.infrastructure.message-handler.tail_request.csv
corfu.infrastructure.message-handler.token_req.csv
corfu.infrastructure.message-handler.trim_mark_request.csv
corfu.infrastructure.message-handler.update_committed_tail.csv
corfu.infrastructure.message-handler.version_request.csv
corfu.infrastructure.message-handler.write.csv
corfu.infrastructure.sequencer.multi-stream-token.csv
corfu.infrastructure.sequencer.query-token.csv
corfu.infrastructure.sequencer.tx-token.csv
corfu.infrastructure.stream-ops.compaction.csv
corfu.runtime.client-router.bootstrap_sequencer.csv
corfu.runtime.client-router.committed_tail_request.csv
corfu.runtime.client-router.compact_request.csv
corfu.runtime.client-router.inspect_addresses_request.csv
corfu.runtime.client-router.keep_alive.csv
corfu.runtime.client-router.known_address_request.csv
corfu.runtime.client-router.layout_bootstrap.csv
corfu.runtime.client-router.layout_committed.csv
corfu.runtime.client-router.layout_prepare.csv
corfu.runtime.client-router.layout_propose.csv
corfu.runtime.client-router.layout_request.csv
corfu.runtime.client-router.log_address_space_request.csv
corfu.runtime.client-router.management_failure_detected.csv
corfu.runtime.client-router.management_healing_detected.csv
corfu.runtime.client-router.node_state_request.csv
corfu.runtime.client-router.orchestrator_request.csv
corfu.runtime.client-router.ping.csv
corfu.runtime.client-router.prefix_trim.csv
corfu.runtime.client-router.range_write.csv
corfu.runtime.client-router.read_request.csv
corfu.runtime.client-router.reset_logunit.csv
corfu.runtime.client-router.seal.csv
corfu.runtime.client-router.sequencer_metrics_request.csv
corfu.runtime.client-router.tail_request.csv
corfu.runtime.client-router.token_req.csv
corfu.runtime.client-router.trim_mark_request.csv
corfu.runtime.client-router.update_committed_tail.csv
corfu.runtime.client-router.version_request.csv
corfu.runtime.client-router.write.csv
corfu.runtime.log-unit-client.10.112.74.235:9000-read.csv
corfu.runtime.log-unit-client.10.112.74.236:9000-read.csv
corfu.runtime.log-unit-client.10.112.74.237:9000-read.csv

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

The .csv files are metrics created from the corfu DB, metrics are enabled by default in NSX-T 3.x.

Resolution

Upgrade to NSX-T 3.2.2 or later to resolve this issue. 

Workaround:

1. Remove all the corfu****.csv files to relieve the storage consumption 
2. Following which, an upgrade will need to be scheduled to fix the issue.