VoyenceControl: Jobs and tasks stuck in "queued for execution" state

Products

• VMware

Issue/Introduction

Symptoms:




Jobs and tasks stuck in "queued for execution" state.
Device Server is running on a VM slice.

SSH to the Device Server does not work.

Device server is pingable.

Telnet to the Device Server over port 443 is successful.

SSH connections made by syssyncd to the Device Server are sustained, stalling at an ESTABLISHED state.

Tasks are sitting on the Application Server waiting to be synchronized to the Device Server.

Environment

VMware Smart Assurance - NCM

Cause

There is a configuration issue with the VM slice which results in this lockup.  The issue can be further confirmed by executing "netstat -anp | grep syssync" on the application server.  It will show sustained connections with a status of "ESTABLISHED" for the IP address of the problematic device server.  Normally, these are very transient connections.

Resolution

The VM configuration will need to be examined and corrected to prevent this issue. The issue is sporadic, and can be temporarily remedied by restarting the problematic device server, but will also require the Voyence service be restarted on the Application Server:

• On the Application Server, restart the Voyence service:
  
  /etc/init.d/Voyence restart