All NetOps Portal reports return query failure IDs

Products

• Network Observability
• CA Performance Management

Issue/Introduction

All reporting is down, not able to query any data

For any report getting "Query failed. See logs for details." with a Query ID which is diff based on the specific report panel.

I am seeing vertica errors in karaf.out on the DA

System Status shows everything is Green or normal. Nothing appears broken in the Portal web UI except for the failing reports. Other views like Groups, SNMP Profiles, Users and Portal MySql based pages work fine.

The console PCService.log and device manager DMService.log files show no errors despite the UI failures.

When we enable detailed view logging and reproduce the problem in the PCService.log we see this error message.

ERROR | Model-Invoker-16         | 2025-06-13 10:51:53,046 | com.ca.im.portal.plugins.rib.models.RIBMultiChartModel
      | RIB query failed
          Model ID/Type: {1000281/RIBMultiChartModel}
          Result status: {FAILED}
          Query ID: {b33e9058-8949-4d04-9948-8ac38bed71db}
          RIB source URL: {https://<DA_HostName>:8582/rib}
          Query: {SELECT .PollItem.ID, .PollItem.DevName, .PollItem.DeviceID, .Item.Name, .BitsPerSecondIn.Avg FROM CA.IM.DA.MF.NormalizedPortInfo.IFSTATS WHERE (.PollItem.ID = 6310963 OR .PollItem.ID = 6310964) AND .EndTime(300) > 1749811860 AND .EndTime(300) <= 1749826260 GROUPBY .PollItem.ID, .PollItem.DevName, .PollItem.DeviceID, .Item.Name ORDERBY .BitsPerSecondIn.Avg DESC LIMIT 48}
          Reason: {{401 Unauthorized: "{<EOL>"servlet":"RestServlet",<EOL>"message":"Unauthorized",<EOL>"url":"/rib/query",<EOL>"status":"401"<EOL>}"}
Caused by:
{401 Unauthorized: "{<EOL>"servlet":"RestServlet",<EOL>"message":"Unauthorized",<EOL>"url":"/rib/query",<EOL>"status":"401"<EOL>}"}}

Checking the SsoEncyrptionDecryptionKey value on the DA at the /rest/dataaggregator URL, and checking Portal via the SsoConfig commands SSO settings, we see matching values. which all match. We see as an example this from the DA:

• <SsoEncryptionDecryptionKey>Z6_fI9=cZ4=eX6@v</SsoEncryptionDecryptionKey>

We see this from Portal SsoConfig (option 1 for DX NetOps -> option 4 for Single Sign-On).

• Encryption Decryption Key: Z6_fI9=cZ4=eX6@v

We confirmed the sysetm default admin user is not using an expired password and the user is enabled.

No errors are observed in DA logs files and a restart of the DA (including run of "dadaemon clean" from the scripts directory) showed no change to the problem.

Environment

All supported Network Observability DX NetOps Performance Management Portal web server releases

Cause

Undetermined at this time.

Resolution

Restart the Portal web server services.

See the Restart NetOps Portal documentation topic for additional details.

Additional Information

If seen please open a new support case along with CARE logging packages from the Portal web server and the DA it connects with.

Steps to run the re.sh script to generate a CARE logging package is found here.