Data Aggregator processes starts but port 8581 or 8582 is not listening

Products

• Network Observability
• CA Performance Management

Issue/Introduction

After patching, the Data Aggregator (DA), Data Collector (DC), and Data Repository (DR) all show as red and failed in the NetOps Portal GUI System Status page.

The Data Aggregator shows Synchronization Failure in the NetOps Portal GUI.

Attempted the following:

• systemctl stop dadaemon
• systemctl stop activemq
  • verified if stopped
• <DA_HOME>/scripts/dadaemon clean
• systemctl start dadaemon

dadaemon and activemq processes start fine but DA not listening on port 8581or 8582

 

After what appears to be a clean start, the karaf.log contains the following entries that repeat endlessly:

Creating Service {http://netqos.com/nqevents/EventManager}IEventManagerWSService

Failed to push ## events: Datasource not synced

Batch process job: GroupsETLJob completed

Batch process job: GlobalItemIdETLJob completed

Batch process job: PollItemsETLJob completed

Cause

Part of configuration was lost;

Check if the file <KARAF_HOME>/etc/org.ops4j.pax.web.cfg is empty.

Resolution

• Stop the DA
  • systemctl stop dadaemon
  • systemctl stop activemq
• update the file <KARAF_HOME>/etc/org.ops4j.pax.web.cfg
  • HTTP:
    

    org.osgi.service.http.enabled=true
    org.osgi.service.http.secure.enabled=false
    org.osgi.service.http.port=8581
    org.ops4j.pax.web.config.file=${karaf.base}/etc/jetty.xml

  • HTTPS:
    

    org.osgi.service.http.enabled=false
    org.osgi.service.http.secure.enabled=true
    org.osgi.service.http.port.secure=8582
    org.ops4j.pax.web.config.file=${karaf.base}/etc/jetty.xml

• start the DA.

Option: You could always use sslConfig.sh to configure for HTTP, then rerun to configure for HTTPS. It should rebuild the .cfg files correctly.

Additional Information

In one of the instances of this issue, the karaf.out log reported :

11:19:34.802 [EclipseGeminiBlueprintExtenderThread-1] ERROR org.ops4j.pax.web.service.internal.HttpServiceStarted - Could not start the servlet context for context path []
java.io.IOException: Failed to bind to /0.0.0.0:8080
...
Caused by: java.net.BindException: Address already in use

...

11:19:37.260 [ItemRepository-thread-11] ERROR com.ca.im.dm.pollcfgctlr.impl.PollerConfigControllerRepositoryListener - Cannot determine normalized facet or VC for component 6180 on device 8

...

11:19:38.674 [FullSyncRequestConsumer-thread-1] ERROR com.broadcom.netops.da.kafka.consumers.GenericConsumer - Unable to create consumer as kafka bootstrap server is not configured
11:19:38.677 [OnDemandItemStatusConsumer-thread-1] ERROR com.broadcom.netops.da.kafka.consumers.GenericConsumer - Unable to create consumer as kafka bootstrap server is not configured
11:19:42.514 [EventPush] ERROR com.ca.im.connector.eventproducer.EventProducerWS - Failed to push 13 events: Datasource not synced