oi_connector fails to ingest data to DX O2 SaaS - Persistent HTTP 404 (Not Found) on All Logical REST Endpoints
search cancel

oi_connector fails to ingest data to DX O2 SaaS - Persistent HTTP 404 (Not Found) on All Logical REST Endpoints

book

Article ID: 423571

calendar_today

Updated On:

Products

DX Unified Infrastructure Management (Nimsoft / UIM) CA Unified Infrastructure Management On-Premise (Nimsoft / UIM) CA Unified Infrastructure Management SaaS (Nimsoft / UIM) DX SaaS DX OI SaaS DX Operational Intelligence DX Operational Observability

Issue/Introduction

We are unable to complete the Northbound integration (UIM  DX O2 SaaS) because the DX O2 API Gateway rejects all known REST ingestion endpoints with a HTTP 404 Not Found error, even when using the correct host and a valid token. The oi_connector log repeatedly shows "Nass is unavailable" due to connection failure.

Related issues:
 
First issue - the customer encountered a connection error as per the oi_connector log:
[main, oi_connector] java.lang.IllegalArgumentException: Data Engine connection string fails to contain database provider.
 at com.nimsoft.nimbus.lookup.DataEngineLookup.parseConnectionString(DataEngineLookup.java:263)
 at com.nimsoft.nimbus.lookup.DataEngineLookup.getConnectionStringsFromDataEngine(DataEngineLookup.java:139)
 at com.nimsoft.nimbus.lookup.DataEngineLookup.getConnectionStringsFromDataEngine(DataEngineLookup.java:94)
 at com.nimsoft.probe.gateway.oi_connector.utils.AXAUtils.getConnectionString(AXAUtils.java:1261)
 at com.nimsoft.probe.gateway.oi_connector.GlobalConfiguration.getConnectionInfo(GlobalConfiguration.java:103)
 at com.nimsoft.probe.gateway.oi_connector.GlobalConfiguration.init(GlobalConfiguration.java:70)
 at com.nimsoft.probe.gateway.oi_connector.OIConnectorProbe.doStart(OIConnectorProbe.java:84)
 at com.nimsoft.probe.gateway.oi_connector.OIConnectorProbe.main(OIConnectorProbe.java:61)
Controller: Max. restarts reached for probe 'oi_connector' (command = <startup java>)
 
Second issue - Jarvis is unavailable error.
[attach_clientsession, oi_connector] Retaining Data in the QoS Queue, as Nass is Unavailable.
[QUEUE_MONITOR_THREAD, oi_connector] Jarvis is unavailable. Alarm, Inventory and Group Queues status is not monitored
[QUEUE_MONITOR_THREAD, oi_connector] Nass is unavailable.

and

[DOI_MONITOR_THREAD, oi_connector] Exception in validateResponseFromJarvis : org.apache.http.conn.HttpHostConnectException: Connect to <saas_url>:443 [<saas_url>/xx.xx.xx.xx] failed: Connection timed out: getsockopt

and

[DOI_MONITOR_THREAD, oi_connector] Jarvis is Down. unavailable
[DOI_MONITOR_THREAD, oi_connector] Sending a Critical Alarm, as Jarvis URL is Down!!
[DOI_MONITOR_THREAD, oi_connector] Jarvis is Down. Counter value : 2264
[DOI_MONITOR_THREAD, oi_connector] Starting nass monitoring
[DOI_MONITOR_THREAD, oi_connector] Inside validateResponseFromNASS()
[DOI_MONITOR_THREAD, oi_connector] URL provided :: https://<saas_url>/metadata/registerMetric
[DOI_MONITOR_THREAD, oi_connector] payload for validateResponseFromNASS : {"metrics":{"sourceName":"dummySource","type":1,"attributeName":"dummyAttribute","attributes":{"product":"UIM"}}]}
[attach_clientsession, oi_connector] Rejecting Data from the QoS Queue, as the Queue Retention period is passed 
[QUEUE_MONITOR_THREAD, oi_connector] Checking the status of the Alarm queue subscription...
[QUEUE_MONITOR_THREAD, oi_connector] Alarm Subscription is ok......
[QUEUE_MONITOR_THREAD, oi_connector] Nass is unavailable.
Third issue - proxy setup and required credentials.

Environment

Environment:

  • oi_connector v2.03
    • proxy setup
  • apm_bridge v2.03
The connector and bridge listed above requires UIM 23..4 CU 5 or higher. Note that in CU5, Java has been updated to Java 21.
  • DX UIM On Prem v23.4 CU0 (23.4.0)
  • DX O2 SaaS v25.1
  • oi_connector v2.03 (note this version requires Java 21)
  • apm_bridge: 2.02/2.03
  • OS: Windows 2019 - Primary Hub doesn't have Internet access so cannot connect to portal
  • oi_connector, apm_bridge - were originally running on the Primary hub
    • oi_connector, subsequently deployed on a UIM Robot which has Internet connectivity - to connect to DX O2 SaaS

Cause

  • HTTP 404 Not Found error was most likely caused by the misconfiguration as well as the proxy not being setup/working as expected
  • DX O2 SaaS/oi_connector configuration
  • oi_connector.cfg contained leftover artifacts from a previous/older version install
  • General misconfiguration
  • Proxy misconfiguration

Resolution

oi_connector requirements

Deployment:

   oi_connector deployed on a robot connected to the Primary hub

   oi_connector robot access to the DX SaaS Portal (Internet)

   oi_connector deployment on Secondary Hub Robot: add key, connect_to_primary_hub = true


Data Engine: data_engine_address needs to be configured using the NimBUS address to connect to the backend UIM DB.

Robot Security:  Ensure the robot security folder and certificate.pem file within it, is also present.


All of the requirements for oi_connector configuration are also fully explained in the techdocs so this must be reviewed.

OI Connector AC Configuration

Steps to resolve the reported issues:

  1. Downgraded oi_connector v2.03 to 2.02. (since 2.03 is only supported as of DX UIM 23.4 CU5 or higher

  2. Replaced the Tenant ID string with the DX O2 SaaS cohort id (obtained via the DX O2 'Connector Parameters' tile/page)

  3. Updated proxy configuration but when the connection was tested it threw an error. Note that the customer required use of a proxy server to connect to the SaaS instance (same for oi_connector)

"One or more fields have failed validation."

In oi_connector v2.02 proxy credentials are mandatory. In 2.03 this has been fixed.

So with oi_connector v2.02, there is a workaround for this requirement:

  1. Enter some 'dummy' credentials
  2. Click Save
  3. Deactivate oi_connector
  4. Edit the oi_connector.cfg
  5. Remove the proxy credentials (username and password)
  6. Save the file
  7. Activate oi_connector

Once the final oi_connector configuration was adjusted as described above, the probe began operating as expected and no longer failed with Max. restarts.

See example oi_connector.log output below:

[QOS_PROCESSOR_THREAD-3, oi_connector] Metric to CS context not available in DB, skipping metric of size: 455
[QosExecutorService-7, oi_connector] QOS PERF:  Total Processing Time(ms): 0 size: 500
[QOS_PROCESSOR_THREAD-7, oi_connector] Received QOS PDS of Size: 500
[QOS_PROCESSOR_THREAD-7, oi_connector] CI Metric cache miss met_id count ::498
[QOS_PROCESSOR_THREAD-7, oi_connector] Loading data from DB for CI Metric cache
[QOS_PROCESSOR_THREAD-7, oi_connector] PERF getCiInformationByMetIdSet(): time(ms) : 0 size:498
[QOS_PROCESSOR_THREAD-7, oi_connector] Loading data from DB for CI Metric cache Done(ms): 0
[QOS_PROCESSOR_THREAD-7, oi_connector] Metric to CS context not available in DB, skipping metric of size: 498
[QosExecutorService-11, oi_connector] QOS PERF:  Total Processing Time(ms): 0 size: 500

Note that these are not errors. This is info that the cache has not been filled yet.

At this point, in general, its important to check to make sure that the QOS and alarms are making it into the backend UIM database by carefully checking the data_engine.log.

Additional Information

Via Raw configure, you MUST confirm the settings for the oi_connector probe on the given robot as documented i the techdocs:

connect_to_primary_hub = true
primary_hub_address <value>
data_engine_address <value>

Also make sure the certificate.pem is in place and configured on the robot as per the oi_connector techdocs.

Once the oi_connector starts, then access the oi_connector configuration in the Admin Console and confirm UIM groups and origins are displayed.

Double check and confirm the DX O2 configuration values AND the proxy configuration, e.g., check Tenant ID (cohort ID) which in this case was not set with the correct value.

If the proxy settings are wrong, you will see connection errors in the log. Double check the proxy configuration and correct it to continue making progress.

If proxy is mandated then every request whether from browser or an application only routes via the proxy. (in this case the proxy was not properly configured). If you can confirm that opening a browser from the machine where the oi_connector probe is running and you are able to access the DXO2 (assuming proxy is configured via browser as well), then the same proxy should ideally work via oi_connector as well.

If there is no proxy involved in the communication (you can check the browser Developer tools-> Network tab) to see where the response is coming from. In that case, then no proxy configuration is required in the oi_connector.

Other (proxy)

The workaround for the mandatory fields for proxy username and password is:

Provide some dummy values for username & password for the proxy.

Open the oi_connector in Raw configure mode.

Remove any values in:

   username (resource/properties/proxy_user)

   password (resource/properties/proxy_password)

...fields.

Click Save to save the configuration.

The probe will be restarted and uses proxy without  user authentication.

Powered by Wolken Software