Paging directory utilization on one of the VMware Cloud Director (VCD) cells is frequently reaching very high levels, causing the root partition to fill up

Products

• VMware Cloud Director

Issue/Introduction

This issue was observed as the root partition filling up on one of the VMware Cloud Director cells, specifically due to growth in the ActiveMQ paging storage.

The large usage was seen under the directory: /opt/vmware/vclouddirector/data/activemq/defaultBroker/data

A snapshot of the directory shows:

2.1M  bindings  
221M  journal  
4.0K  largemessages  
25G   paging

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log indicate:

AMQ222038: Starting paging on address '$.artemis.internal.sf.vcd-cluster.<id>'; globalSize=5########### bytes; globalMaxSize=5############## bytes;
AMQ224108: Stopped paging on address 'Backend-activityEvents'; ... 
AMQ224108: Stopped paging on address '$.artemis.internal.sf.vcd-cluster.<id>'; ...

Environment

VMware Cloud Director 10.x

Cause

ActiveMQ Artemis enters paging mode when the memory limit for message storage is exceeded and needs to offload messages to disk. The disk files generated are page files that store messages on disk and should be removed once messages are fully consumed or acknowledged.

Resolution

You can proceed with removing the paging files from the following directory using the rm command, for example:

rm ###.page

Directory path:

/opt/vmware/vcloud-director/data/activemq/defaultBroker/data/paging/

Example file listing:

-rw------- 1 vcloud vcloud   10 Nov 24 06:47 address.txt  
-rw------- 1 vcloud vcloud   10M Dec 10 05:57 0000####.page  
-rw------- 1 vcloud vcloud   10M Dec 10 05:59 0000####.page  
-rw------- 1 vcloud vcloud   10M Dec 10 06:00 0000####.page  
-rw------- 1 vcloud vcloud  9.9M Dec 10 06:02 0000####.page  
-rw------- 1 vcloud vcloud  2.7M Dec 10 06:02 0000#####.page

Additionally, as a workaround, you can temporarily disable paging on the affected cell by running:

cell-management-tool manage-config -n jms.persistenceEnabled -v false

This setting will prevent further paging file creation, which helps stop the paging directory from filling up the root partition. Temporarily disabling paging means that persistent paging behavior is turned off, so new paging files will no longer accumulate on disk while this setting is in effect.

Additional Information

In normal operation, ActiveMQ Artemis removes page files as messages are acknowledged and fully processed. However, if broker internal addresses (e.g., cluster messaging channels) are paged and not correctly acknowledged or if paging is triggered repeatedly without successful cleanup, the files persist. This is consistent with observed behavior where large paging directories fill disk and do not clear between broker restarts.