Why is the TC Server instance getting restarted automatically in production environment

Products

VMware Tanzu Spring Runtime

Issue/Introduction

We found the TC Server instance restarted itself in Windows 2022 production environment. How can we identify the root cause of restarting issue?

Environment

Windows TC Server 4.1.30

Cause

The error entries in wrapper log indicating the tcserver instance (JVM) managed by the wrapper on Windows 2022 encountered a critical issue, leading to its termination and restarting.

INFO | jvm 1 | 2025/09/17 10:52:51 | Origin Name - http://#######

ERROR | wrapper | 2025/09/17 10:53:25 | JVM appears hung: Timed out waiting for signal from JVM.

STATUS | wrapper | 2025/09/17 10:53:25 | Dumping JVM state.

ERROR | wrapper | 2025/09/17 10:53:29 | JVM did not exit on request, terminated

STATUS | wrapper | 2025/09/17 10:53:34 | Launching a JVM...

INFO | jvm 2 | 2025/09/17 10:53:40 | INFO: APM Insight javaagent v7.4.0 successfully hooked up with JVM (PID:####). For more information check agent logs

located at D:\APMInsight\PIJR

INFO | jvm 2 | 2025/09/17 10:53:40 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org

INFO | jvm 2 | 2025/09/17 10:53:40 |

INFO | jvm 2 | 2025/09/17 10:53:42 | Sep 17, 2025 10:53:42 AM org.apache.catalina.startup.Catalina load

INFO | jvm 2 | 2025/09/17 10:53:42 | INFO: Server initialization in [1317] milliseconds

INFO | jvm 2 | 2025/09/17 10:53:54 | FwUtil: ThreadPool Initialized. Thread Pool Size: 30

Reason/explanation of the log entries:

ERROR | wrapper | 2025/09/17 10:53:25 | JVM appears hung: Timed out waiting for signal from JVM
- The Java Service Wrapper, which is designed to monitor and manage the JVM process, did not receive a timely response or "signal" from the JVM. This suggests the JVM process became unresponsive or entered a state where it could no longer communicate effectively with the Wrapper.
STATUS | wrapper | 2025/09/17 10:53:25 | Dumping JVM state:
- The Wrapper initiated a process to capture the current state of the JVM. This "dump" often includes thread dumps, heap dumps, or other diagnostic information that can be crucial for post-mortem analysis and identifying the root cause of the hang.
ERROR | wrapper | 2025/09/17 10:53:29 | JVM did not exit on request, terminated:
- Since the JVM was unresponsive and did not exit gracefully after the Wrapper's request, the Wrapper was forced to forcefully terminate the JVM process. This is often done using a SIGKILL signal on Unix-like systems or an equivalent termination mechanism on Windows.
STATUS | wrapper | 2025/09/17 10:53:34 | Launching a JVM...
- After terminating the unresponsive JVM, the Wrapper automatically attempted to restart the JVM process to restore the application's functionality.

Some potential Causes of a hung (unresponsive) JVM (tcserver instance):

Resource Exhaustion: The JVM might have exhausted system resources like memory (leading to OutOfMemoryErrors) or CPU, causing it to become unresponsive.
Deadlocks or Infinite Loops: The application running within the JVM could have entered a deadlock state or an infinite loop, preventing it from processing requests or sending signals back to the Wrapper.

Resolution

It is considered normal for a Tanuki Service Wrapper to restart the TC Server instance (JVM) process became hung and unresponsive.

For further troubleshooting to determining why the JVM hung:

Gather the thread and heap dumps
Analyze the Thread Dump for:
- Deadlocks: Threads waiting for resources held by other threads, forming a circular dependency.
- Blocked Threads: Threads perpetually waiting for a monitor that is never notified.
- Long-running operations: Threads stuck in computationally intensive tasks or I/O operations.
- Infinite loops: Threads repeatedly executing a block of code without termination.
Analyze the heap:
- Use tools like Eclipse Memory Analyzer to analyze the heap dump.