Troubleshooting NSX disk space related issues

Products

VMware NSX

Issue/Introduction

This article provides a reference to knowledge articles about addressing high disk space utilization levels on NSX nodes

Environment

VMware NSX

Resolution

/var/log/cbm/
- cbm.log - Cluster boot manager log, responsible for forming cluster.
/var/log/cloudnet/
- nsx-ccp.log - Controller updates log.
/var/log/cm-inventory
- cm-inventory.log
/var/log/core/ - Core dumps
/var/log/corfu/
- corfu.9000.log - Corfu transaction logs
- corfu-compactor-audit.log - Logs details for corfu compaction runs.
/var/log/migration-coordinator - v2t migration logs
/var/log/nvpapi - Appliance API logs
/var/log/phonehome-coordinator - Alarms and alerts
/var/log/proton/
- nsxapi.log - Main manager log.

/var/log/proxy/
- envoy.log - Envoy service log
- envoy_access.log - Lists API calls to the manager
- reverse-proxy.log - Details about external API calls which are proxied internally
/var/log/search - UI search logs
/var/log/stats - Ping, I/O stats, etc
/var/log/upgrade-coordinator - Upgrade logs
/var/log/vmware - CPU and memory stats

Other manager directories of note

/image/core - Stores core dumps from services when they crash
/repository/ - Stores configuration files, such as vibs for hosts, etc, do not edit/delete these files, unless under Broadcom guidance.

NSX edge node log locations

/var/log/syslog - Catch all logging
/var/log/frr - Routing information
/var/log/lb - Load balancer logs
/var/log/core - Core dumps
/var/log/nsx-cli - Local nsxcli commands run
/var/log/nvpapi - Local node api calls

ESXi host log locations

/var/run/log/esxupdate.log - VIB, install, upgrade operations
/var/run/log/hostd - Host attachments info
/var/run/log/vmkernel.log - Dataplane issues
/var/run/log/dfwpktlogs.log - Firewall rule logging
/var/run/log/nsx-nest-db - Nestdb datastore logging
/var/run/log/nsx-syslog.log - NSX catch all log
/var/run/log/nsxcli.log - NSX cli operations

NSXCLI

Using the nsxcli, logs can be viewed using the commands:

get log-file <>

The below options are available for the command:

auth.log - Authorization log
controller - Controller log
controller-error - Controller error log
http.log - HTTP service log
kern.log - Kernel log
manager.log - Manager service log
node-mgmt.log - Node management log
nsx-audit-write.log - NSX audit write log
nsx-audit.log - NSX audit log
syslog - System log

Commands

The command can also be appended by 'follow', for example:

get log-file syslog follow

To check disk usage:

As admin user:

get filesystem-stats

As root user:

df -h

To check for large files, commands such as 'du' can be used as root user, for example:

du -sh -> will display the size of the current directory in human readable format.