Secondary hubs are not communicating with the Primary hub

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

We are not able to discover secondary hub server in UIM console.

Network connectivity is working fine between Primary hub and secondary and tunnel hub servers, but the secondary hub and tunnel server is not visible in the console.

Environment

• Release: 20.3

Cause

• queued baseline messages caused no space left on filesystem

Resolution

Secondary hub was showing as red in IM.

Since there was no telnet installed on the secondary hub we ran-> ncat <hub_hostname> 48002

Secondary hub IP in the Primary hub shows as expected.

We will check the following once we can login to the secondary hub machine and try contacting the Primary:

ping <secondary_hub_machine>
tracert <secondary_hub_machine>
ncat <secondary_hub_machine> 48002

Checked disk as well using df -k and found that the root filesystem was full.

Commands run:

   du . --max-depth=1

   find / -type f -size +500M

An additional symptom presented itself, e.g., no space left on / filesystem-> and we couldn't login using a known/good userid.

   find /opt/xxx/nimsoft -type f -size +100M

Cause-> baseline QOS messages were so queued up that there was no space left on the / file system

1. We freed up space by deleting the baseline QOS messages using rm -rf *.sds
2. Then we restarted the robot and it turned green in the IM and the queue cleared up as well.