Hub tunnel communications between primary and secondary were being reset. This was preventing discovery_server and _agent communications from completing. This was showing up in the hub and discovery logs with permission and access denied messages for the record_agent_status command.
The following is found in the secondary hubs' discovery_agent log showing the permission denied.
[StatusUpdaterOnConfig, discovery_agent] Encountered error updating Discovery Agent status to Discovery Server (/Domain/PrimaryHub/Robot/discovery_server): (6) permission denied, Received status (6) on response (for sendRcv) for cmd = 'record_agent_status'.
[StatusUpdaterOnConfig, discovery_agent] Error trying to send status: PDS [ht={probe_version=Element [buf=null, pds=null, tpds=null, type=7, value=8.51], state=Element [buf=null, pds=null, tpds=null, type=7, value=NO_SCHEDULE], agent_address=Element [buf=null, pds=null, tpds=null, type=7, value=/Domain/SecondaryHub/Robot/discovery_agent]}]
[StatusUpdaterOnConfig, discovery_agent] (6) permission denied, Received status (6) on response (for sendRcv) for cmd = 'record_agent_status'
The following is found in the primary hub log:
hub: taccess_lock - taccess_check
hub: taccess_check: - [1] ###.###.###.###/53479 /<Domain>/<PrimaryHub>/<Robot>/discovery_server record_agent_status discovery_agent
hub: taccess_unlock - taccess_check
The taccess messages in the primary hub are referencing the Access List found in the hub tunnel configuration. When these access lists are configured, tunnel communications can be blocked.