vSAN traces daemon (vsantraced service) fails with admission failure errors
search cancel

vSAN traces daemon (vsantraced service) fails with admission failure errors

book

Article ID: 399290

calendar_today

Updated On:

Products

VMware vSAN 8.x

Issue/Introduction

This article helps identify and resolve when a host is hitting a memory leak for vsantraced service.

One of more of below symptoms help diagnose this issue:

  • ESXI host may show as in 'not Responding state'
  • vsantraced service is not running
  • /var/run/log/vmkernel.log of a host shows messages similar to:

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu36:79390030)cartelheap.79390030 (222917398) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu36:79390030)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79390030:cartelheap.79390030
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu36:79390030)cartelheap.79390030 (222917398) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:79389893)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79389893:cartelheap.79389893
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:79389893)cartelheap.79389893 (222916512) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:79389893)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79389893:cartelheap.79389893
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu23:79389893)cartelheap.79389893 (222916512) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.

  • Issue is caused by insufficient memory to be allocated by vsantraced daemon.
  • vsantraced.log shows messages similar to:

YYYY-MM-DDTHH:MM:SS Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory
YYYY-MM-DDTHH:MM:SS Er(11) chardevlogger[329373891]: Failed to read trace:Cannot allocate memory
YYYY-MM-DDTHH:MM:SS Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory
YYYY-MM-DDTHH:MM:SS Er(11) chardevlogger[329373891]: Failed to read trace:Cannot allocate memory
YYYY-MM-DDTHH:MM:SS Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory

Environment

VMware vSphere ESXi 8.0.2

Cause

A memory leak has been identified in the vsantraced service on ESXi 8.0 U2. This issue is isolated to the vSAN management stack.

Resolution

If issues with the vsantraced service occur, follow these steps to restart the daemon or modify the compression thread settings.

Step 1:
The initial troubleshooting step involves restarting the service on each affected node to resolve transient execution errors.

  1. Log in to the ESXi host via SSH as root.

  2. Execute the following command:
    /etc/init.d/vsantraced restart


Step 2:
If the issue persists after the service restart, the Zstandard (ZSTD) compression threads may require adjustment to improve stability.

  1. Open the configuration file using the vi editor:
    vi /etc/vmware/vsan/vsantraced.conf

  2. Locate the VSANTRACED_ZSTD_THREAD entry.

  3. Update the value to 1:
    VSANTRACED_ZSTD_THREAD=1

Note: If the setting is not present, add the following line to the file: VSANTRACED_ZSTD_THREAD=1

Step 3:
To activate the modified configuration, the daemon must be restarted.

  1. Execute the restart command:
    /etc/init.d/vsantraced restart

Powered by Wolken Software