After upgrading CA Performance Management, DA or DC processes appear to be running, but may not be initialized

Products

CA Infrastructure Management CA Performance Management Network Observability

Issue/Introduction

The DA or DC may not be initialized due to karaf failure after upgrade or after removal of karaf data directory

Symptoms:

On certain systems, the DA or DC processes start successfully and the status shows that the process is running, but the DA or DC application is never initialized. This has typically been seen with underpowered CPU’s that lead to slower application initialization, and only when the DA or DC are upgraded, or restarted after clearing out the karaf data directory (which can happen automatically after an ungraceful shutdown).
Checking the apache-karaf-*/data/log/karaf.log after the restart shows an error message like:

2019-04-08 22:57:57,106 | WARN | Event Dispatcher | AetherBasedResolver | mvn.internal.AetherBasedResolver 583 | 3 - org.ops4j.pax.logging.pax-logging-api - 1.8.3 | | Error resolving artifactcom.ca.im:data-mgmt.provision:xml:features:3.1.0-RELEASE-207:Could not find artifact com.ca.im:data-mgmt.provision:xml:features:3.1.0-RELEASE-207
shaded.org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.ca.im:data-mgmt.provision:xml:features:3.1.0-RELEASE-207

at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
at shaded.org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
…
OR

java.io.IOException: Error resolving artifact com.ca.im:data-mgmt.provision:xml:features:2.7.0-RELEASE-137: Could not find artifact com.ca.im:data-mgmt.provision:xml:features:2.7.0-RELEASE-137 in central (http://repo1.maven.org/maven2/)

at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:584)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:528)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:506)
at org.ops4j.pax.url.mvn.internal.AetherBasedResolver.resolve(AetherBasedResolver.java:481)
at org.ops4j.pax.url.mvn.internal.Connection.getInputStream(Connection.java:123)

Environment

CAPM 2.7 and later

Cause

Logging into the karaf console through ssh can confirm that the DA or DC features are not resolved within karaf.

On DA: ssh –p8501 karaf@localhost
On DC: ssh –p8601 karaf@localhost
- features:list | grep data
- shows no feature named “data-aggregator” or “data-collection-manager”

Connecting to karaf and manually resolving the features through karaf commands will initialize the applications. If the DA failed to initialize then the DC’s will need to be restarted as well (but it is not necessary to apply the workaround on the DC unless the same error messages have been seen in the karaf.log file).

Please note that these workarounds may have to re-applied if the apache-karaf-XXX/data directory is removed / cleaned up again. This can typically occur if:

a) The application is upgraded OR
b) The application is restarted after an ungraceful shutdown.

Resolution

Workaround Steps for the DA:
1. Identify the features repository URL to be added from the INSTALL_ROOT/apache-karaf-2.3.0/etc/org.apache.karaf.features.cfg file.

[root@host ~]# grep --color provision /opt/IMDataAggregator/apache-karaf-2.4.3/etc/org.apache.karaf.features.cfg

featuresRepositories = mvn:org.apache.karaf.assemblies.features/standard/2.4.3/xml/features,mvn:org.apache.karaf.assemblies.features/spring/2.4.3/xml/features,mvn:org.apache.karaf.assemblies.features/enterprise/2.4.3/xml/features,mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features

#featuresRepositories=mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features

NOTE: The content that you need to select from this file will line up with the error message that is seen in the karaf.log file. Also note that depending on the version of CAPM, the number after RELEASE- may change while the number before it will reflect the release of CAPM in use.

2. Connect through karaf and manually run the commands needed to add the features:

> ssh –p8501 karaf@localhost
Password: DXN3t0ps (or check the correct password in /opt/IMDataAggregator/apache-karaf/etc/users.properties file)
> features:repo-add mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features
> features:install data-aggregator

3. Check the karaf.log, and you will start seeing the various bundles in the application initializing, and should not see a repeat of the “Error resolving artifact” message.

4. Check that application has initialized (for example, check the DA_HOST:8581/rest/ page)

Workaround Steps for the DC:
1. Identify the features repository URL to be added from the INSTALL_ROOT/apache-karaf-2.3.0/etc/org.apache.karaf.features.cfg file.

[root@host ~]# grep --color provision /opt/IMDataCollector/apache-karaf-2.4.3/etc/org.apache.karaf.features.cfg

featuresRepositories = mvn:org.apache.karaf.assemblies.features/standard/2.4.3/xml/features,mvn:org.apache.karaf.assemblies.features/spring/2.4.3/xml/features,mvn:org.apache.karaf.assemblies.features/enterprise/2.4.3/xml/features,mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features

#featuresRepositories=mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features

Again, similar to the DA above, the content that you need to select from this file will line up with the error message that is seen in the karaf.log file. And again, depending on the version of CAPM, the number after RELEASE- may change.

2. Connect through karaf and manually run the commands needed to add the features:

> ssh –p8601 karaf@localhost
Password: DXN3t0ps
> features:addurl mvn:com.ca.im/data-mgmt.provision/2.7.0-RELEASE-133/xml/features
> features:install data-collection-manager

3. Check the karaf.log, and you will start seeing the various bundles in the application initializing, and should not see a repeat of the “Error resolving artifact” message.

4. Check that application has initialized (for example, check the System Status > Data Collectors page and ensure that the DC status says ‘Collecting Data’)

Alternatively: Reinstalling DA and DC over the top, it re-laying down the jar files that may be corrupt and can get you back up and running.

Additional Information

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/performance-management/24-3/troubleshooting.html