Devices show "DC Connection Lost" as a status and missing DC Host field value
Article ID: 386200
Updated On:
Products
Issue/Introduction
The customer changed the PM IP addresses and the day after upgraded PM. After a few days, they noticed the majority of the devices (around 1500) have the DC Connection Lost status. Only 40 devices are being monitored.
The Data Collector is up and running:
Environment
PM: Any version
Cause
The DC Host field is missing / empty.
The device was assigned to DC 5351. But the DC REST only showed one DC 2590333
DC 5351 must have been deleted.
DC 2590333 was created at Sun Jan 12 14:54:33 2025. That was the date/time they upgraded PM.
http://DA-Host:8581/rest/dcms
<DataCollectionMgrInfoList><DataCollectionMgrInfo version="1.0.0"><ID>2590333</ID><DcmID>DC_host:2387b6c8-e753-4866-9b1d-aef0414c1035</DcmID><Enabled>true</Enabled><IsStandby>false</IsStandby><IPAddress>xx.yy.ww.zz</IPAddress><RelatedDeviceItem>2590334</RelatedDeviceItem><ProtocolCollectorStateGroupList><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.health.collector.DCMHealthCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.icmp.collector.IcmpCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.telemetry.collector.TelemetryCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.mediation.platform.impl.MediationSimpleCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.sdn.performance.impl.SDNCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup><ProtocolCollectorStateGroup><ProtocolCollectorName>com.ca.im.dm.snmp.collector.SnmpCollector</ProtocolCollectorName><ProtocolCollectorState>true</ProtocolCollectorState></ProtocolCollectorStateGroup></ProtocolCollectorStateGroupList><HostName>DC_host</HostName><MonitoredDeviceCount>40</MonitoredDeviceCount><CollectorState>RUNNING</CollectorState><IPAddressType>4</IPAddressType><DCIPDomainID>2</DCIPDomainID><TenantName>Default Tenant</TenantName><IsAlso><IsA name="Versions" rootURL="versions" /><IsA name="DeviceComponent" rootURL="devices/components" /><IsA name="IPDomainMember" rootURL="ipdomainmember" /><IsA name="DataCollectorPollingInfo" rootURL="dcpolling" /><IsA name="DistributedItemRepositoryInfo" rootURL="distireps" /><IsA name="ConnectivityMonitoringInfo" rootURL="connectivitymonitoring" /><IsA name="Pollable" rootURL="pollable" /></IsAlso><NormalizedDCMHealthInfo version="1.0.0"><SourceFacetTypesList><SourceFacetTypes>{http://im.ca.com/certifications/dcmp}DCMHealthInfo</SourceFacetTypes></SourceFacetTypesList><NRM_PolledItemCount>5935</NRM_PolledItemCount></NormalizedDCMHealthInfo><IPDomainMember version="1.0.0"><IPDomainID>2</IPDomainID></IPDomainMember><ConnectivityMonitoringInfo version="1.0.0"><ConnectivityAdminStatus>NOT_INSTALLED</ConnectivityAdminStatus><ConnectivityOperStatus>NOT_INSTALLED</ConnectivityOperStatus></ConnectivityMonitoringInfo><DistributedItemRepositoryInfo version="1.0.0"><Status>OK</Status><LastStateChangeTime>Fri Jan 17 10:42:29 2025 +0000</LastStateChangeTime></DistributedItemRepositoryInfo><Item version="1.0.0"><Name>DC_host - DCM0</Name><CreateTime>Sun Jan 12 14:54:33 2025 +0000</CreateTime><MDRItemID>2590333</MDRItemID></Item><DataCollectorPollingInfo version="1.0.0"><LastStateChangeTime>Fri Jan 17 10:42:29 2025 +0000</LastStateChangeTime><Status>OK</Status></DataCollectorPollingInfo><Versions version="1.0.0"><ComponentVersionsList><ComponentVersions><ComponentNames>Data Collector</ComponentNames><MajorVersions>23</MajorVersions><MinorVersions>3</MinorVersions><PatchVersions>11</PatchVersions><BuildNumbers>5</BuildNumbers></ComponentVersions></ComponentVersionsList></Versions></DataCollectionMgrInfo></DataCollectionMgrInfoList>
http://DA-Host:8581/rest/devices/1231285
<Device version="1.0.0"><ID>1231285</ID><ContactStatus>UP</ContactStatus><PrimaryIPAddress>xx.yy.ww.zz</PrimaryIPAddress><PolledItemCount>1</PolledItemCount><StopLoading>false</StopLoading><SupportedProtocolsList><SupportedProtocols>ICMP</SupportedProtocols></SupportedProtocolsList><DiscProfileID>18075</DiscProfileID><supportsOnDemandMFDiscovery>true</supportsOnDemandMFDiscovery><CalculatedContactStatus>DC_CONNECTION_LOST</CalculatedContactStatus><StopPoll>false</StopPoll><RelatesTo><MonitoredGroupIDList relatesURL="relatesto/monitoredgroups" rootURL="monitoredgroups"><ID>5197</ID></MonitoredGroupIDList><DiscoveryProfileIDList relatesURL="relatesto/discoveryprofiles" rootURL="discoveryprofiles"><ID>18075</ID></DiscoveryProfileIDList><GroupIDList relatesURL="relatesto/groups" rootURL="groups"><ID>9441</ID><ID>5299</ID><ID>5197</ID><ID>700725</ID><ID>5280</ID></GroupIDList></RelatesTo><IsAlso><IsA name="IPDomainMember" rootURL="ipdomainmember" /><IsA name="Lifecycle" rootURL="lifecycle" /><IsA name="AccessibleDevice" rootURL="devices/accessible" /><IsA name="Syncable" rootURL="syncable" /><IsA name="MetricFamilyDiscoveryHistory" rootURL="devices/mfdiscoveryhistory" /><IsA name="Pollable" rootURL="pollable" /></IsAlso><Syncable version="1.0.0"><SyncID>328433</SyncID></Syncable><DeviceMonitoringProfile version="1.0.0"><ConsolidatedMonitoringProfile>5400</ConsolidatedMonitoringProfile></DeviceMonitoringProfile><IPDomainMember version="1.0.0"><IPDomainID>2</IPDomainID></IPDomainMember><Lifecycle version="1.0.0"><TimeStamp>Thu Jan 1 0:00:00 1970 +0000</TimeStamp><State>ACTIVE</State></Lifecycle><Item version="1.0.0"><AlternateName>gvemstsprof001</AlternateName><Name>xx.yy.ww.zz</Name><CreateTime>Wed Oct 25 13:19:46 2023 +0000</CreateTime><MDRItemID>5351</MDRItemID></Item></Device>
at:
java.base/java.lang.Thread.getStackTrace(Unknown Source)
com.ca.im.dm.core.database.item.vertica.impl.ItemDbInterfaceImpl.logDeletes(ItemDbInterfaceImpl.java:1543)
com.ca.im.dm.core.database.item.vertica.impl.ItemDbInterfaceImpl.deleteItems(ItemDbInterfaceImpl.java:1506)
jdk.internal.reflect.GeneratedMethodAccessor493.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base/java.lang.reflect.Method.invoke(Unknown Source)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
jdk.proxy32/jdk.proxy32.$Proxy129.deleteItems(Unknown Source)
com.ca.im.item.impl.repository.ItemOperations.deleteItems(ItemOperations.java:1005)
com.ca.im.item.impl.repository.ItemRepositoryImpl.deleteItems(ItemRepositoryImpl.java:592)
com.ca.im.item.impl.repository.ItemRepositoryImpl.deleteItem(ItemRepositoryImpl.java:569)
com.ca.im.item.ws.impl.WebServiceEndPointImpl.delete(WebServiceEndPointImpl.java:734)
com.ca.im.item.ws.impl.AutoWebServiceHandlerImpl.deleteByIDWithTenant(AutoWebServiceHandlerImpl.java:978)
com.ca.im.item.ws.impl.AutoWebServiceHandlerImpl.deleteByID(AutoWebServiceHandlerImpl.java:965)
Resolution
Run the following DA REST to update the DC Host field of each device:
curl -u admin -H 'Content-Type: application/xml' -X PUT -d '<Device version="1.0.0"><Item version="1.0.0"><MDRItemID>XXXXX</MDRItemID></Item></Device>' http://DA:8581/rest/devices/<Device itemid>
Example:
curl -u admin -H 'Content-Type: application/xml' -X PUT -d '<Device version="1.0.0"><Item version="1.0.0"><MDRItemID>2590333</MDRItemID></Item></Device>' http://DA:8581/rest/devices/1231285
Here are the detailed steps to update the DC Host field (on all devices with DC Lost Contact status) in bulk:
a) In the Portal host, run the following command line to create the output.csv file with the ItemIDs with Contact Status = DC Connection Lost status:
mysql -h localhost -u root -p -e 'select LocalID from netqosportal.dst_device where SourceID IN(select SourceID from netqosportal.data_sources2 where SourceType=0x40000) and dst_device.ContactStatus REGEXP "DC_CONNECTION_LOST"' | sed 's/\r$//' > /tmp/output.csv
b) Edit the /tmp/output.csv file and remove the first line (LocalID) which is the column name and save the file.
c) In the Portal host, create a script file (example: /tmp/changedchost.sh) with the following entries:
#!/bin/sh
for i in `cat output.csv`; do
echo "curl -u "admin:<password>" --header 'Content-Type: application/xml' --request PUT --data '<Device version="1.0.0"><Item version="1.0.0"><MDRItemID>2590333</MDRItemID></Item></Device>' --url http://<DA_host>:8581/rest/devices/${i}" >> /tmp/changed_devices.out
curl -u "admin:<password>" --header 'Content-Type: application/xml' --request PUT --data '<Device version="1.0.0"><Item version="1.0.0"><MDRItemID>2590333</MDRItemID></Item></Device>' --url http://<DA_host>:8581/rest/devices/${i}
sleep 1
echo " " >> /tmp/changed_devices.out
done;
d) In the above script, replace the following fields: <password> <DA_host>
Note that 2590333 is the current DC ID.
e) Set file permission to:
chmod 775 changedc.sh
f) Run the script:
./changedchost.sh
g) Review the output file: /tmp/changed_devices.out
h) Run DA (Data Aggregator) Full Resynchronization.
Additional Information
Review the output.csv file in case you get this message while running the changedchost.sh script:
curl: (3) Illegal characters found in URL
cat -v output.csv
Ensure there is no special characters like ^M
Feedback
