NSX metrics consume a large disk space on NSX Manager nodes

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• You are running NSX-T 3.x
• /var/log/corfu is consuming a large about of space

1.7G /var/log/cbm
2.5G /var/log/cloudnet
8.8G /var/log/corfu
2.5G /var/log/journal
2.0G /var/log/proton
1.8G /var/log/vmware

• Files similar to the below are observed in /var/log/corfu

corfu.infrastructure.message-handler.bootstrap_sequencer.csv
corfu.infrastructure.message-handler.committed_tail_request.csv
corfu.infrastructure.message-handler.compact_request.csv
corfu.infrastructure.message-handler.failure_detector_metrics_request.csv
corfu.infrastructure.message-handler.flush_cache.csv
corfu.infrastructure.message-handler.inspect_addresses_request.csv
corfu.infrastructure.message-handler.keep_alive.csv
corfu.infrastructure.message-handler.known_address_request.csv
corfu.infrastructure.message-handler.layout_bootstrap.csv
corfu.infrastructure.message-handler.layout_committed.csv
corfu.infrastructure.message-handler.layout_prepare.csv
corfu.infrastructure.message-handler.layout_propose.csv
corfu.infrastructure.message-handler.layout_request.csv
corfu.infrastructure.message-handler.log_address_space_request.csv
corfu.infrastructure.message-handler.management_bootstrap_request.csv
corfu.infrastructure.message-handler.management_failure_detected.csv
corfu.infrastructure.message-handler.management_healing_detected.csv
corfu.infrastructure.message-handler.management_layout_request.csv
corfu.infrastructure.message-handler.node_state_request.csv
corfu.infrastructure.message-handler.orchestrator_request.csv
corfu.infrastructure.message-handler.ping.csv
corfu.infrastructure.message-handler.prefix_trim.csv
corfu.infrastructure.message-handler.range_write.csv
corfu.infrastructure.message-handler.read_request.csv
corfu.infrastructure.message-handler.reset.csv
corfu.infrastructure.message-handler.reset_logunit.csv
corfu.infrastructure.message-handler.restart.csv
corfu.infrastructure.message-handler.seal.csv
corfu.infrastructure.message-handler.sequencer_metrics_request.csv
corfu.infrastructure.message-handler.sequencer_trim_req.csv
corfu.infrastructure.message-handler.streams_address_request.csv
corfu.infrastructure.message-handler.tail_request.csv
corfu.infrastructure.message-handler.token_req.csv
corfu.infrastructure.message-handler.trim_mark_request.csv
corfu.infrastructure.message-handler.update_committed_tail.csv
corfu.infrastructure.message-handler.version_request.csv
corfu.infrastructure.message-handler.write.csv
corfu.infrastructure.sequencer.multi-stream-token.csv
corfu.infrastructure.sequencer.query-token.csv
corfu.infrastructure.sequencer.tx-token.csv
corfu.infrastructure.stream-ops.compaction.csv
corfu.runtime.client-router.bootstrap_sequencer.csv
corfu.runtime.client-router.committed_tail_request.csv
corfu.runtime.client-router.compact_request.csv
corfu.runtime.client-router.inspect_addresses_request.csv
corfu.runtime.client-router.keep_alive.csv
corfu.runtime.client-router.known_address_request.csv
corfu.runtime.client-router.layout_bootstrap.csv
corfu.runtime.client-router.layout_committed.csv
corfu.runtime.client-router.layout_prepare.csv
corfu.runtime.client-router.layout_propose.csv
corfu.runtime.client-router.layout_request.csv
corfu.runtime.client-router.log_address_space_request.csv
corfu.runtime.client-router.management_failure_detected.csv
corfu.runtime.client-router.management_healing_detected.csv
corfu.runtime.client-router.node_state_request.csv
corfu.runtime.client-router.orchestrator_request.csv
corfu.runtime.client-router.ping.csv
corfu.runtime.client-router.prefix_trim.csv
corfu.runtime.client-router.range_write.csv
corfu.runtime.client-router.read_request.csv
corfu.runtime.client-router.reset_logunit.csv
corfu.runtime.client-router.seal.csv
corfu.runtime.client-router.sequencer_metrics_request.csv
corfu.runtime.client-router.tail_request.csv
corfu.runtime.client-router.token_req.csv
corfu.runtime.client-router.trim_mark_request.csv
corfu.runtime.client-router.update_committed_tail.csv
corfu.runtime.client-router.version_request.csv
corfu.runtime.client-router.write.csv
corfu.runtime.log-unit-client.10.112.74.235:9000-read.csv
corfu.runtime.log-unit-client.10.112.74.236:9000-read.csv
corfu.runtime.log-unit-client.10.112.74.237:9000-read.csv

Environment

VMware NSX-T Data Center 3.x

Cause

The .csv files are metrics created from the corfu DB, metrics are enabled by default in NSX-T 3.x.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

1. Remove all the corfu****.csv files to relieve the storage consumption 
2. Following which, an upgrade will need to be scheduled to fix the issue.