Troubleshooting & debugging Palo Alto Datasource in Aria Operations for Networks
Article ID: 337853
Updated On:
Products
Issue/Introduction
This article provides information about Troubleshooting & debugging Palo Alto Networking Devices.
Environment
Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.12.1
Aria Operations for Networks 6.13.0
Aria Operations for Networks 6.14.0
Cause
This article provides information about Troubleshooting & debugging Palo Alto Networking Devices in Aria Operations for Networks when adding a Palo Alto data source.
Resolution
Adding Palo Alto Networking Device in vRNI:
-
Required User Privileges to add Palo Alto Data Source:
Create a new admin for data source addition, an admin with admin role Superuser will work but create a user with custom admin role:
Disable Web UI Access, CLI Access, from XML API tab only allow configurational and operational access. Create a user with the custom role and use while adding data source in VRNI.
- Logs to troubleshoot when adding PAN device fails / timed out
~/logs/collector
Error in Validate Credentials PAN logs in Collector:
ubuntu@proxy-infra:~/logs/collector$ grep -nr "PANConnectionEntity" .| grep "Error" ./collector.STDOUT-2017-10-31-10.24.56.log.error:437782:ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-10.24.56.log.error:438782:ERROR [2017-11-01 14:15:07,766] [U:379,F:057,T:436,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-10.24.56.log.error:441157:ERROR [2017-11-01 14:16:34,881] [U:348,F:098,T:446,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-20.05.36.log:697333:ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121]
ERROR [2017-11-01 14:12:15,406] [U:338,F:080,T:418,M:2,121] dataprovider.utils.HttpUtils:[HttpUtils:checkCodeAndThrow:49] - [Thread-1] - Could not get response for /api, status 403 ERROR [2017-11-01 14:12:15,408] [U:338,F:080,T:418,M:2,121] dataprovider.utils.HttpUtils:[HttpUtils:checkStatusAndThrow:36] - [Thread-1] - API /api error response <response status = 'error' code = '403'><result><msg>User not authorized to perform this operation.</msg></result></response> ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121] pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan com.vnera.dataproviders.core.common.impl.dataprovider.utils.exceptions.HttpException: Could not get response for /api, status 403 at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkCodeAndThrow(HttpUtils.java:50) at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkStatusAndThrow(HttpUtils.java:29) at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkStatusAndThrow(HttpUtils.java:18) at com.vnera.dataproviders.core.impl.pan.utils.PanCommonUtils.fetchFromPAN(PanCommonUtils.java:90) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.fetchFromPAN(PANUtils.java:1426) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.fetchAPIResp(PANUtils.java:1417) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.getManagerStatus(PANUtils.java:205) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.testConnection(PANUtils.java:197) at com.vnera.dataproviders.core.impl.pan.southbound.PANConnectionEntity.hasPrivileges(PANConnectionEntity.java:87) at com.vnera.dataproviders.core.common.DataProviderFactory.validateCredentials(DataProviderFactory.java:413) at com.vnera.collector.core.engine.SaasCommandProcessor.processMessage(SaasCommandProcessor.java:187) at com.vnera.collector.core.saascommunication.SaasListener.receiveMessage(SaasListener.java:76) at com.vnera.collector.externalcommunication.saascommunication.SaasRPCListener.sendResponse(SaasRPCListener.java:55) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.receiveMessage(AbstractSaasListenerTransport.java:129) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.lambda$run$0(AbstractSaasListenerTransport.java:175) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:73) at com.github.rholder.retry.Retryer.call(Retryer.java:104) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.run(AbstractSaasListenerTransport.java:162) at java.lang.Thread.run(Thread.java:748)
~/logs/restapilayer INFO [2017-11-01 17:35:56] c.v.r.CustomerResource:[?:?:?] - [dw-841 - POST /management/validateCredentials] - getTransport(11740, PAN) INFO [2017-11-01 17:35:56] c.v.r.SaaSCommunicationHelper:[?:?:?] - [dw-841 - POST /management/validateCredentials] - getSaaSServiceClient(11740, IOYL3BZ) INFO [2017-11-01 17:35:56] c.v.r.SaaSCommunicationHelper:[?:?:?] - [dw-841 - POST /management/validateCredentials] - Got IP from KeyVal:XX.XX.XX.XX INFO [2017-11-01 17:35:59] c.v.r.c.v.r.VneraBackendService:[?:?:?] - [dw-841 - POST /management/validateCredentials] - validateCredential took:2944
~/logs/collector WARN [2017-11-01 17:38:20,787] [U:253,F:194,T:446,M:2,121] core.common.DataProviderFactory:[DataProviderFactory:validateCredentials:281] - [Thread-1] - Connection validation for PAN initiated with config: _collectorId:IOYL3BZ PAN_URL:https://XX.XX.XX.XX:443 PAN_USER:api PAN_PWD:******* nickName: notes: dpId: ENCRYPTED_CONFIG:true
INFO [2017-11-01 17:40:49,215] [U:349,F:091,T:440,M:2,121] core.common.DataProviderFactory:[DataProviderFactory:createNewDataProvider:131] - [Thread-1] - Creating data provider PAN_XX.XX.XX.XX for customer 11740 with config: _collectorId:IOYL3BZ nickName:pan8 notes: dpId: lastModifiedTimestamp:XXXXXXXXXXXXX ENCRYPTED_CONFIG:true lastActivityTimestamp:XXXXXXXXXXXXX dpState:ACTIVE PAN_URL:https://XX.XX.XX.XX:443 PAN_USER:api PAN_PWD:******* INFO [2017-11-01 17:40:49,215] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:74] - [Thread-1] - Data provider initialization started. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:79] - [Thread-1] - Config Validation succeeded. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:84] - [Thread-1] - DP operations manager successfully initiated. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[Tasker:runTask:429] - [Thread-1] - Running 25: com.vnera.dataproviders.core.common.impl.dataprovider.AbstractDataProvider$InitializationAsyncTask_1 INFO [2017-11-01 17:40:49,229] [U:349,F:091,T:440,M:2,121] 197.17.11}:[Tasker$TaskerCallBack:<init>:111] - [Thread-1] - Creating callback 25: TaskContext {taskType=ASYNC, taskIdentifier=com.vnera.dataproviders.core.common.impl.dataprovider.AbstractDataProvider$InitializationAsyncTask_1} INFO [2017-11-01 17:40:49,230] [U:350,F:090,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:93] - [Thread-1] - DP initialization done. WARN [2017-11-01 17:40:49,230] [U:350,F:090,T:440,M:2,121] core.dataprovidermanager.DataProviderManager:[DataProviderManager:startDataProvider:404] - [Thread-1] - INFO: DP instance started: PAN_1XX.XX.XX.XX for customerId XXXXX
- CURL request to verify user creds validation:
Successful response:
<response status = 'success'><result><key>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</key></result></response>
curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=<API_KEY>&cmd=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>" curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>"
Successful response:
<response status="success"><result><sync_status nsx_mgr_id="15"><last_dynamic_update>12:38AM Oct 05 2017</last_dynamic_update><status></status><sync_info>Registered</sync_info></sync_status> </result></response>
Error response:
curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&cmd=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>" <response status = 'error' code = '403'><result><msg>User not authorized to perform this operation.</msg></result></response>
curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=?type=op&action=show&key=$ {API_KEY} &cmd=<request><partner><vmware-service-manager><status></status></vmware-service-manager></partner></request>" In case of error, cross check user created in panorama has valid XML API access.
grep -nr "PANUtils" . ./collector.STDOUT-2017-10-31-22.06.42.log.error:392692:INFO [2017-11-01 11:32:56,565] [U:319,F:151,T:470,M:2,121] pan.utils.PANUtils:[PANUtils:init:145] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Init started ./collector.STDOUT-2017-10-31-22.06.42.log.error:392709:INFO [2017-11-01 11:32:58,857] [U:243,F:258,T:500,M:2,121] pan.utils.PANUtils:[PANUtils:init:166] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Init completed grep -nr "PANUtils" . | grep "Exception" grep -nr "PANUtils" . | grep "ERROR"
For Devices related parsing and fetch:
grep -nr "PanDevicesCommandParser" . collector.STDOUT-2017-11-01-06.57.08.log:178805:INFO [2017-11-01 09:45:43,335] [U:337,F:077,T:414,M:2,121] pan.parsers.PanDevicesCommandParser:[PANDeviceGroupCommandParser:getDeviceDetails:29] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - No devices found with deviceGroup name shared collector.STDOUT-2017-11-01-06.57.08.log:185571:ERROR [2017-11-01 09:50:43,610] [U:172,F:270,T:442,M:2,121] pan.parsers.PanDevicesCommandParser:[PanPhysicalHelper:parseVirtualSystemsAndZones:304] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Failed to parse VirtualSystems And Zones collector.STDOUT-2017-11-01-06.57.08.log:185592:INFO [2017-11-01 09:50:43,610] [U:172,F:270,T:442,M:2,121] pan.parsers.PanDevicesCommandParser grep -nr "PanDevicesCommandParser" . | grep "Exception" grep -nr "PanDevicesCommandParser" . | grep "ERROR"
sudo su cd /var/log/hadoop-yarn/containers/application_<>/ --------------------------------------------------------------------------------------- grep -nr "PANFirewallStoreConfigProgram" . | grep "Exception" grep -nr "PANServiceStoreConfigProgram" . | grep "Exception" grep -nr "PANManagerStoreConfigProgram" . | grep "Exception" grep -nr "PANDeviceGroupStoreConfigProgram" . | grep "Exception" grep -nr "PANDeviceStoreConfigProgram" . | grep "Exception" grep -nr "ANLogicalInterfaceStoreConfigProgram". | grep "Exception" grep -nr "PANVirtualRouterStoreConfigProgram" . | grep "Exception" grep -nr "PANVirtualSystemStoreConfigProgram" . | grep "Exception" grep -nr "PANZoneStoreConfigProgram". | grep "Exception"
To validate if platform received specific pan message type sdm, do following:
Collecting SDM dumps for engineering analysis
Validate in vRNI proxy, if Collector is sending message to Platform:
grep -nr "sending message key:" | grep "com.vnera.model.pan.Addressbase.config" collector.STDOUT-2017-10-31-22.06.42.log.error:263270:INFO [2017-11-01 07:24:42,903] [U:277,F:179,T:456,M:2,121] dataprovider.utils.CollectorUtils$PayloadCache:[CollectorUtils:pushPayLoadToCollector:281] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - sending message key:hashKey:https://XX.XX.XX.XX:443, upstreamDataType=com.vnera.model.pan.Addressbase.config, consumerIdentifier=1, dpIdentifier=11740_PAN_XX.XX.XX.XX payloadHash:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx size:7779 type:com.vnera.model.pan.Addressbase.config hash:https://XX.XX.XX.XX:443 collector.STDOUT-2017-10-31-22.06.42.log.error:392710:INFO [2017-11-01 11:32:58,985] [U:342,F:158,T:500,M:2,121] dataprovider.utils.CollectorUtils$PayloadCache:[CollectorUtils:pushPayLoadToCollector:281] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - sending message key:hashKey:https://XX.XX.XX.XX:443, upstreamDataType=com.vnera.model.pan.Addressbase.config, consumerIdentifier=1, dpIdentifier=11740_PAN_XX.XX.XX.XX payloadHash:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx size:7779 type:com.vnera.model.pan.Addressbase.config hash:https://XX.XX.XX.XX:443
create folder to collect dump:
mkdir -p /home/ubuntu/sdm-dump/
Run following KafkaReader command to dump sdms of specific message type, replace value of messageType param accordingly:
java -cp ~/build-target/common-utils/tools-0.001-SNAPSHOT.jar com.vnera.tools.KafkaReader 0 Topic3 -1 localhost 9092 0 -messageType com.vnera.model.pan.Addressbase.config -dumpSDMsToFile /home/ubuntu/sdm-dump/pan
--------------------------------------
com.vnera.model.pan.DeviceGroup.config
com.vnera.model.pan.Addressbase.config
com.vnera.model.pan.Servicebase.config
com.vnera.model.pan.Firewall.config
com.vnera.model.pan.PANManager.config
com.vnera.model.pan.PhysicalDevice.config
com.vnera.model.pan.NSXVMSeriesDevice.config
com.vnera.model.pan.LogicalDevice.config
com.vnera.model.pan.VirtualRouter.config
com.vnera.model.pan.VirtualSystem.config
com.vnera.model.pan.Zone.config
com.vnera.model.pan.LogicalInterface.config
Feedback
