In test of failover, HA probe was not able to start neither the queues or probes with a permission denied error:
All queues / probes throw the same kind of errors.
Jan 23 10:16:41:347 1 HA: INFO: activating queue: prediction_engine.PREDICTION_CONFIG
Jan 23 10:16:41:347 2 HA: INFO: 'activate' queue via 'queue_active_callback' callback.
Jan 23 10:16:41:347 4 HA: CONNECT: 0000000000B5CE60(364) 10.xx.xxx.xxx/61481->10.xx.xxx.xxx/48002
Jan 23 10:16:41:347 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:41:347 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=364
Jan 23 10:16:41:347 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:41:347 2 HA: WARN: hub queue_active failed to 'activate' queue 'prediction_engine.PREDICTION_CONFIG', nRetryCounter: 0. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:42:354 4 HA: CONNECT: 0000000000B5CE60(420) 10.xx.xxx.xxx/61484->10.88.xxx.xxx/48002
Jan 23 10:16:42:354 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:42:354 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=420
Jan 23 10:16:42:354 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:42:354 2 HA: WARN: hub queue_active failed to 'activate' queue 'prediction_engine.PREDICTION_CONFIG', nRetryCounter: 1. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:43:357 4 HA: CONNECT: 0000000000B5CE60(420) 10.xx.xxx.xxx/61493->10.88.xxx.xxx/48002
Jan 23 10:16:43:357 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:43:357 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=420
Jan 23 10:16:43:357 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:43:357 2 HA: WARN: hub queue_active failed to 'activate' queue 'prediction_engine.PREDICTION_CONFIG', nRetryCounter: 2. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:44:362 4 HA: CONNECT: 0000000000B5CE60(364) 10.xx.xxx.xxx/61495->10.88.xxx.xxx/48002
Jan 23 10:16:44:362 1 HA: INFO: activating queue: probeDiscovery
Jan 23 10:16:44:362 2 HA: INFO: 'activate' queue via 'queue_active_callback' callback.
Jan 23 10:16:44:362 4 HA: CONNECT: 0000000000B5CE60(364) 10.xx.xxx.xxx/61497->10.xx.xxx.xxx/48002
Jan 23 10:16:44:362 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:44:362 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=364
Jan 23 10:16:44:362 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:44:362 2 HA: WARN: hub queue_active failed to 'activate' queue 'probeDiscovery', nRetryCounter: 0. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:45:379 4 HA: CONNECT: 0000000000B5CE60(364) 10.xx.xxx.xxx/61498->10.xx.xxx.xxx/48002
Jan 23 10:16:45:379 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:45:379 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=364
Jan 23 10:16:45:379 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:45:379 2 HA: WARN: hub queue_active failed to 'activate' queue 'probeDiscovery', nRetryCounter: 1. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:46:395 4 HA: CONNECT: 0000000000B5CE60(420) 10.xx.xxx.xxx/61500->10.xx.xxx.xxx/48002
Jan 23 10:16:46:395 3 HA: SREQUEST: queue_active ->10.xx.xxx.xxx/48002
Jan 23 10:16:46:395 3 HA: RREPLY: status=permission denied(6) <-10.xx.xxx.xxx/48002 h=37 d=0 fd=420
Jan 23 10:16:46:395 3 HA: SREQUEST: _close ->10.xx.xxx.xxx/48002
Jan 23 10:16:46:395 2 HA: WARN: hub queue_active failed to 'activate' queue 'probeDiscovery', nRetryCounter: 2. Retrying after a sleep cycle. rc: 6, error: permission denied
Jan 23 10:16:47:412 4 HA: CONNECT: 0000000000B60FD0(432) 10.xx.xxx.xxx/61501->10.xx.xxx.xxx/48000
Release : 20.x/20/3.x
Component : UIM - HA
- missing probe security permissions
In IM under Security-> Probe->Administration make sure you see the HA probe listed with:
Probe, Access and IP Mask as:
HA admin *
and if not, try redeploying the HA probe and check again.