Error: "unable to create native thread: possibly out of memory or process/resource limits reached" seen in vco-app logs accompanying Orchestrator crash
search cancel

Error: "unable to create native thread: possibly out of memory or process/resource limits reached" seen in vco-app logs accompanying Orchestrator crash

book

Article ID: 443324

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Possible symptoms:

  • Workflows start failing unexpectedly in Aria Orchestrator.
  • Users may face an error when logging in to Orchestrator GUI: 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'. 
  • Errors such as the following seen in /services-logs/prelude/vco-app/file-logs/vco-server-app.log :
    • Failed to init context with package com.vmware.vim.binding.cis.license : Class com.vmware.vim.binding.cis.license.Asset already registered
    • ERROR vco [host='vco-app-####' thread='licenseComplianceScheduler-1' user='' org='' trace=''] {} org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task
      java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
  • Verification: This particular issue is due to exhausted thread counts. This can be monitored using the following command:
    • tail -f -n 10000 /services-logs/prelude/vco-app/file-logs/vco-server-app_metrics.log | grep 'type=GAUGE, name=count, value='
    • A figure in the thousands is cause for concern / likely confirms the issue.
    • Once this count goes over 21000 (threads), the limit may soon be reached, crashing the Orchestrator pod

Environment

  • VMware Aria Orchestrator (external) 8.18.1, unpatched or with patch 1-5
  • Using VMware vCenter Server for authentication

Cause

There is a known issue with the license checking mechanism in Orchestrator,

Resolution

This is a known issue which will be fixed in an upcoming patch release of Aria Orchestrator.

 

Workaround

It is possible to reduce the frequency of checking the license, until a permanent fix is available:

  • vracli vro properties set --key com.vmware.o11n.license.task.interval.wait.default --value 86000

It will still check for license on each Orchestrator (vco-app pod) restart, and every 30s as normal when there's any error, so the impact is minimal.

Powered by Wolken Software