Error PAM-CM-4096: Exception in CustomConnectorUtil.generateRestCall URL http://localhost:9099/capamef/targetConnectors/?type=ALL

Products

• CA Privileged Access Manager (PAM)

Issue/Introduction

During operation of PAM the following error message starts to appear regularly in the session logs

PAM-CM-4096: Exception in CustomConnectorUtil.generateRestCall URL http://localhost:9099/capamef/targetConnectors/?type=ALL. Message was: Connection refused (Connection refused)

however there is no custom connector defined for this machine

At the same time, trying to access the grafana interface results in an error, and syslog messages are not being received

Environment

CA PAM versions 4.1.X and above

Cause

This is caused by the container network not being up: whenever PAM starts, so should start its container network, and each one of the containers will be listening on a different port

This is the internal docker ps from a working PAM appliance

root@<example>:~# docker ps
CONTAINER ID   IMAGE                                                                          COMMAND                  CREATED        STATUS        PORTS                                                                           NAMES
a8b2caef3c50   symantec/victoria-metrics-server:1.71.0-bookworm-20250221                      "/usr/local/bin/vict…"   2 months ago   Up 2 months   127.0.0.1:9100->8428/tcp, 172.17.0.1:9100->                          8428/tcp   victoria-metrics
cac6406925e1   symantec/tcf:4.21.2.12                                                         "catalina.sh run"        2 months ago   Up 2 months   127.0.0.1:9099->8080/tcp                                                        tcf
408e0961c499   symantec/logstash:7.16.3.pam12-bookworm-20250221-temurin-jre-11.0.26.0.0-4-1   "/bin/sh -c /usr/sha…"   2 months ago   Up 2 months   127.0.0.1:9108->9108/udp                                                        logstash
29b4fe2f91a4   symantec/k8v2-cluster-orchestrator:4.2.2.9                                     "java -Djdk.tls.clie…"   2 months ago   Up 2 months   127.0.0.1:9094-9095->9094-9095/tcp                                              k8v2-cluster-orchestrator
67a1dcb26a7a   symantec/k8-cluster-orchestrator:4.2.2.9                                       "java -Djdk.tls.clie…"   2 months ago   Up 2 months   127.0.0.1:9092-9093->9092-9093/tcp                                              k8-cluster-orchestrator
71ec7c1dbe1b   symantec/grafana:10.1.5-bookworm-20250221                                      "/usr/share/grafana/…"   2 months ago   Up 2 months   127.0.0.1:9030->3000/tcp                                                        grafana

If these containers are not running in PAM or there is a conflict, this issue will be present

Resolution

There may be several reasons why the container network is not running. Please check under Configuration --> Network --> Networking settings

 

1. Missing network configuration. If for some reason the container network configuration is missing (e.g. the entries are blank) that will prevent containers from running
2. Overlapping network configuration with appliances. The default container network is 172.17.0.1/16: if there are devices in PAM defined for addresses in this range this may cause a problem starting the containers. In this case, changing the container network settings to prevent overlap from happening and restarting networking should suffice
3. The docker service is not running inside the appliance for some unknown reason

To get a solution to situations 1 and 3 some help from Broadcom Support to access the appliance and correct the issue

For use case 2 please correct the docker network settings to prevent overlap with devices defined in CA PAM