NCM Device configuration pulls are stuck in "queued for execution"

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

Device servers stopped pulling device configurations, causing jobs to remain stuck in a "Queued for execution" status.

Restarting vcmaster did not resolve the problem.

Cause

An application upgrade performed to 25.4.8 updated Tomcat. Following the upgrade, the Application Server (AS) web server (httpd) began presenting a PKI-signed certificate  instead of the standard NCM self-signed MASTERKEY certificate that the Device Servers trusted.

The syssyncs.log file showed repeated failures (875 times in a single morning) on files like acmd_5004574565_taskresult.xml.

The specific error stated:

SSL certificate verification FAILED: unable to get local issuer certificate (error 20 at depth 0) ... REJECTING non-MASTERKEY certificate due to validation error.

Because of this communication failure, the files remained stuck in the DS retry queue.

Resolution

The resolution required updating the trust store on the Device Servers so they would trust the new PKI-signed certificate being presented by the Application Server.

 

Step 1: Generate the Updated Trust Bundle on the Application Server

On the Application Server, navigate to the tools directory and run the SSL utility script to install the keys and certificates:

Bash

 

cd $VOYENCE_HOME/tools/ssl
perl ssl-utility.pl --install \
/path/to/ca.key \
/path/to/ca.crt \
/path/to/hostname-CA.crt

Note: The administrator must enter the passphrase when prompted. This process generates and updates the trust bundle located at $VOYENCE_HOME/conf/bundle.p12.

 

Step 2: Copy and Apply the Trust Bundle to the Device Servers

The updated bundle.p12 file must be securely copied to each affected Device Server, have its file ownership corrected, and be processed via the local update script.

 

For Device Server (repeat for other DS systems):

 

scp $VOYENCE_HOME/conf/bundle.p12 ncm@hostname:/opt/smarts-ncm/conf/ssh
ssh ncm@hostname "chown ncm:voyence /opt/smarts-ncm/conf/bundle.p12"
ssh ncm@hostname "perl /opt/smarts-ncm/tools/update-ds-bundle.pl"

 

Step 3: Restart Services and Validate

• Service Restart: A voyenced service restart is required on each Device Server after running the script to successfully reload the updated trust bundle.

• Data Flush: Once the servers establish mutual trust, the queued taskresult.xml files auto-flush out of the retry queue; there is no need to manually re-run the configuration pulls.

• Final Status: The customer performed complete system reboots following these steps.