Threat Analytics Health

Products

CA Threat Analytics for PAM CA Privileged Access Manager (PAM)

Issue/Introduction

The health status of CA Threat Analytics applications shows as critical after the reboot of the virtual machine.

Environment

All supported versions of CA Threat Analytics Application.

Cause

The health status was critical due to the number if threads that were being processed were higher that the queue size.

Resolution

1. **Stop** the service.
2. **Kill** any remaining Java processes (`kill -9`).
3. **Fix Permissions** on the `/usr/local/interlock/config/` directory.
4. **Rename** the `standalone/tmp/` and `standalone/data/` folders completely.
5. **Restart** the service and tail the log: `tail -f /path/to/your/server.log`.

Detailed Steps:

### How to Fix:

1. **Hard Kill the Process:** Do not use the standard `stop` script alone. Find the process ID: `ps -ef | grep java` and use `kill -9 <PID>`.
2. **Clean the VFS Cache:** Notice the path in your log: `/jboss/standalone/tmp/vfs/...`. This is a Virtual File System cache. If it gets corrupted or contains "stale" deployment markers, JBoss will throw duplicate service errors.
* Stop the service.
* **Rename the tmp directory:** `mv /usr/local/interlock/gems/gems/torquebox-server-<version>-java/jboss/standalone/tmp/* /usr/local/interlock/gems/gems/torquebox-server-<version>java/jboss/standalone/tmp01/`
* **Rename the data directory:** `mv /usr/local/interlock/gems/gems/torquebox-server-<version>-java/jboss/standalone/data/* /usr/local/interlock/gems/gems/torquebox-server-<version>java/jboss/standalone/data01/`

If doing the above does not help in clearing the markers, follow the steps below:

Clear deployment markers and reset failed state

Login in as interlock or root, before starting/restarting Interlock:

Look for the following marker files and remove them, if exists:
/usr/local/interlock/gems/gems/torquebox-server-<version>-java/jboss/standalone/deployments/interlock-knob.yml.failed
/usr/local/interlock/gems/gems/torquebox-server-<version>-java/jboss/standalone/deployments/interlock-knob.yml.isdeploying

Then

sudo systemctl reset-failed torquebox.service

And then start: sudo systemctl start torquebox

---------

[0m02:39:57,292 INFO [stdout] (pool-1-thread-1) Failed to initialize adapter for CA PAM - Permission denied - /usr/local/interlock/config/nginx/TAPtoPAM.conf.bak

## 2. The Permission Error: `TAPtoPAM.conf.bak`

The log shows:
`Failed to initialize adapter for CA PAM - Permission denied - /usr/local/interlock/config/nginx/TAPtoPAM.conf.bak`

**What this means:**
The Threat Analytics application (Interlock) is trying to read or write to its configuration backup file during the adapter initialization. Because it cannot access this file, the **CA PAM adapter fails to load**.

### How to Fix:

Check the ownership of the config directory. It is likely owned by `root`, but the application is running as a different user (e.g., `interlock` or `jboss`).

Even the directory nginx should have the owner and group as interlock.

* **Run:** `ls -l /usr/local/interlock/config/nginx/TAPtoPAM.conf.bak`
* **Fix:** `chown <app_user>:<app_group> /usr/local/interlock/config/nginx/TAPtoPAM.conf.bak`
* **Broad fix:** `chown -R <app_user>:<app_group> /usr/local/interlock/config/`

---

## Summary of Connection to the 502 Error

1. **Deployment Starts:** JBoss tries to deploy the "Interlock-knob" app.
2. **Crash:** It hits a `DuplicateServiceException` for the `activity_rollup_job`.
3. **Halt:** The web-tier (which handles your browser requests) never finishes starting because the background jobs failed.
4. **Gateway Timeout:** Nginx (the frontend) waits for JBoss to finish booting. When it doesn't happen within ~60 seconds, Nginx gives up and displays the **502 Bad Gateway**.