vSAN traces daemon (vsantraced service) fails with admission failure errors
search cancel

vSAN traces daemon (vsantraced service) fails with admission failure errors

book

Article ID: 399290

calendar_today

Updated On:

Products

VMware vSAN 8.x

Issue/Introduction

This article helps identify and resolve when a host is hitting a memory leak for vsantraced service.


Symptoms:

  • One of more of below symptoms help diagnose this issue:

    • ESXI host may show as in 'not Responding state'
    • vsantraced service is not running
    • /var/run/log/vmkernel.log of a host shows messages similar to:

2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu36:79390030)cartelheap.79390030 (222917398) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu36:79390030)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79390030:cartelheap.79390030
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu36:79390030)cartelheap.79390030 (222917398) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu23:79389893)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79389893:cartelheap.79389893
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu23:79389893)cartelheap.79389893 (222916512) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu23:79389893)Admission failure in path: host/opt/vsan/vsantraced:chardevlogger.79389893:cartelheap.79389893
2025-05-24T08:25:04.000+0100 In(182) vmkernel: cpu23:79389893)cartelheap.79389893 (222916512) requires 260 KB, asked 260 KB from vsantraced (222914968) which has 101356 KB occupied and 20 KB available.

 

Issue is caused by insufficient memory to be allocated by vsantraced daemon.

[vsantraced.log]
2024-11-13T14:45:39.993Z Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory
2024-11-13T14:45:39.993Z Er(11) chardevlogger[329373891]: Failed to read trace:Cannot allocate memory
2024-11-13T14:45:39.993Z Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory
2024-11-13T14:45:39.994Z Er(11) chardevlogger[329373891]: Failed to read trace:Cannot allocate memory
2024-11-13T14:45:39.994Z Er(11) chardevlogger[329373967]: Failed to read trace:Cannot allocate memory

Environment

VMware vSphere ESXi 8.0.2

Cause

Memory leak issue in vsantraced service affecting ESXi 8.0 U2. This issue only affects the vSAN management stack.

Resolution

 

  • User may workaround this problem by restarting vsantraced service. Command to execute on each node experiencing the problem:

/etc/init.d/vsantraced restart

  • if the issue persists after restart of the service user should adjust the value VSANTRACED_ZSTD_THREAD setting inside service's config file. User can edit the configuration file with:

vi /etc/vmware/vsan/vsantraced.conf 

  • and change the value of VSANTRACED_ZSTD_THREAD setting to 1. NOTE: if no such setting is present a new line containing the string:

VSANTRACED_ZSTD_THREAD=1

  • Should be added. Upon saving changes daemon should be reloaded with:

/etc/init.d/vsantraced restart


        

Powered by Wolken Software