Cisco ACI discovery failure with Error : "Exception during discovery : Controller Failed to create block for cisco-aci"

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

Cisco ACI discovery failure with Error : "Exception during discovery : Controller Failed to create block for cisco-aci" and device added in pending list.

Environment

All Supported Smarts Versions

Cause

Basically, as a first step of discovery, Cisco ACI Controller (APIC) is discovered through the ICMP discovery. On the discovery of APIC as a Host in Smart Assurance IP Domain Manager, it communicates with the DCF controller, which provisions a discovery collector. The discovery collector runs REST APIs, which are required to create a topology representation in the Smart Assurance IP domain manager.

If there are no collectors provisioned to discover the hosts, one would expect the reported error message.

Smarts IP Manager Debug Logs, may record the following messages during discovery

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG DEBUG : [Thread-40 ACIControllerUtil]:Creating Collector instance for ACI

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG DEBUG : [Thread-40 RestQueryDispatcher ]:HttpURLConnection reqUrl HTTPS://<DCC API>:<PORT>/dcc/v1/catalog/blocks/cisco-aci/provision method POST body **

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG DEBUG : [Thread-40 RestQueryDispatcher ]: headerKey Authorization value Basic <ENCRYPTED AUTH VALUE>

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG DEBUG : [Thread-40 RestQueryDispatcher ]:SendRequest sun.net.www.protocol.https.DelegateHttpsURLConnection:https://<DCC API>:<PORT>/dcc/v1/catalog/blocks/cisco-aci/provision

[<DATE & TIME>]] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG ERR : [Thread-40 RestQueryDispatcher ]: SendRequest  code 0 responseHeader null errorString null body null

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG ERR : [Thread-40 RestQueryDispatcher ]:Exception Type:java.net.ConnectException Connection refused (Connection refused)
java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)


[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG ERR : [Thread-40 DCCRestClient]:Error occurred while sending request [dcc/v1/catalog/blocks/%s/provision] to the DCC API : [Connection refused (Connection refused)] 

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG ERR : [Thread-40 DiscoverCiscoACI]:Exception occured while discovering the ACI instance : <ACI Device>

[<DATE & TIME>] t@<EPOCH> Discovery #6
DSCV_MSG-*-DSCV_GENERIC-MSG ERR : [Thread-40 DiscoverCiscoACI]:Exception Type:java.lang.Exception Controller Failed to create block for cisco-aci
com.emc.octo.sas.stream.discovery.cisco.aci.ACIControllerUtil.createAndStart(ACIControllerUtil.java:105)
com.emc.octo.sas.stream.discovery.cisco.aci.DiscoverCiscoACI.discoverACI(DiscoverCiscoACI.java:113)

Resolution

• One may confirm the provisioned collectors by running the following:

dmctl -s <AMPM Domain> invoke ICF_PersistentDataSet::ACICollectorIDs get >> CollectorIDs.log

• One may confirm the connections validity by running the following:

curl -u admin:changeme --request GET --url https://<DCC API>:<PORT>/dcc/v1/catalog/blocks/instances
Note: Update the credentials and add --cacert if configured

If there are failures in the above, please review the following

• Whether the access settings are configured correctly. Reference Documentation: Discovery of Cisco ACI
• Whether the port and host connectivity to the DCC is configured correctly or not.

Please engage with Broadcom Support quoting this KB article if the connectivity is fine and yet the errors persist.

Also, if issue is observed on existing device, verify if collectors are running, if stopped, restart it and perform rediscovery of the device.