Troubleshooting & debugging Palo Alto Datasource in Aria Operations for Networks
search cancel

Troubleshooting & debugging Palo Alto Datasource in Aria Operations for Networks

book

Article ID: 337853

calendar_today

Updated On:

Products

VMware Aria Operations for Networks

Issue/Introduction

This article provides information about Troubleshooting & debugging Palo Alto Networking Devices.

Environment

VMware vRealize Network Insight 3.x

Aria Operations for Networks 6.X

Cause

This article provides information about Troubleshooting & debugging Palo Alto Networking Devices in Aria Operations for Networks when adding a Palo Alto data source.

Resolution

Note: IP Address/FQDN of Palo Alto to be used only when adding data source.

Adding Palo Alto Networking Device in vRNI: 

  • Required User Privileges to add Palo Alto Data Source:

Create a new admin for data source addition, an admin with admin role Superuser will work but create a user with custom admin role:

Disable Web UI Access, CLI Access, from XML API tab only allow configurational and operational access. Create a user with the custom role and use while adding data source in VRNI.

  • Logs to troubleshoot when adding PAN device fails / timed out 

~/logs/collector

Error in Validate Credentials PAN logs in Collector:

ubuntu@proxy-infra:~/logs/collector$ grep -nr "PANConnectionEntity" .| grep "Error" ./collector.STDOUT-2017-10-31-10.24.56.log.error:437782:ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-10.24.56.log.error:438782:ERROR [2017-11-01 14:15:07,766] [U:379,F:057,T:436,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-10.24.56.log.error:441157:ERROR [2017-11-01 14:16:34,881] [U:348,F:098,T:446,M:2,121]pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan ./collector.STDOUT-2017-10-31-20.05.36.log:697333:ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121]
 

Error Stacktrace for PAN Data Source Validate Credentials call:

ERROR [2017-11-01 14:12:15,406] [U:338,F:080,T:418,M:2,121] dataprovider.utils.HttpUtils:[HttpUtils:checkCodeAndThrow:49] - [Thread-1] - Could not get response for /api, status 403 ERROR [2017-11-01 14:12:15,408] [U:338,F:080,T:418,M:2,121] dataprovider.utils.HttpUtils:[HttpUtils:checkStatusAndThrow:36] - [Thread-1] - API /api error response <response status = 'error' code = '403'><result><msg>User not authorized to perform this operation.</msg></result></response> ERROR [2017-11-01 14:12:15,409] [U:338,F:080,T:418,M:2,121] pan.southbound.PANConnectionEntity:[PANConnectionEntity:hasPrivileges:91] - [Thread-1] - Error in connecting to pan com.vnera.dataproviders.core.common.impl.dataprovider.utils.exceptions.HttpException: Could not get response for /api, status 403 at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkCodeAndThrow(HttpUtils.java:50) at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkStatusAndThrow(HttpUtils.java:29) at com.vnera.dataproviders.core.common.impl.dataprovider.utils.HttpUtils.checkStatusAndThrow(HttpUtils.java:18) at com.vnera.dataproviders.core.impl.pan.utils.PanCommonUtils.fetchFromPAN(PanCommonUtils.java:90) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.fetchFromPAN(PANUtils.java:1426) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.fetchAPIResp(PANUtils.java:1417) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.getManagerStatus(PANUtils.java:205) at com.vnera.dataproviders.core.impl.pan.utils.PANUtils.testConnection(PANUtils.java:197) at com.vnera.dataproviders.core.impl.pan.southbound.PANConnectionEntity.hasPrivileges(PANConnectionEntity.java:87) at com.vnera.dataproviders.core.common.DataProviderFactory.validateCredentials(DataProviderFactory.java:413) at com.vnera.collector.core.engine.SaasCommandProcessor.processMessage(SaasCommandProcessor.java:187) at com.vnera.collector.core.saascommunication.SaasListener.receiveMessage(SaasListener.java:76) at com.vnera.collector.externalcommunication.saascommunication.SaasRPCListener.sendResponse(SaasRPCListener.java:55) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.receiveMessage(AbstractSaasListenerTransport.java:129) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.lambda$run$0(AbstractSaasListenerTransport.java:175) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:73) at com.github.rholder.retry.Retryer.call(Retryer.java:104) at com.vnera.collector.core.saascommunication.AbstractSaasListenerTransport.run(AbstractSaasListenerTransport.java:162) at java.lang.Thread.run(Thread.java:748)

Valid Credentials PAN logs in Platform:

~/logs/restapilayer INFO [2017-11-01 17:35:56] c.v.r.CustomerResource:[?:?:?] - [dw-841 - POST /management/validateCredentials] - getTransport(11740, PAN) INFO [2017-11-01 17:35:56] c.v.r.SaaSCommunicationHelper:[?:?:?] - [dw-841 - POST /management/validateCredentials] - getSaaSServiceClient(11740, IOYL3BZ) INFO [2017-11-01 17:35:56] c.v.r.SaaSCommunicationHelper:[?:?:?] - [dw-841 - POST /management/validateCredentials] - Got IP from KeyVal:XX.XX.XX.XX INFO [2017-11-01 17:35:59] c.v.r.c.v.r.VneraBackendService:[?:?:?] - [dw-841 - POST /management/validateCredentials] - validateCredential took:2944
 
Validate Credentials logs in Collector:

~/logs/collector WARN [2017-11-01 17:38:20,787] [U:253,F:194,T:446,M:2,121] core.common.DataProviderFactory:[DataProviderFactory:validateCredentials:281] - [Thread-1] - Connection validation for PAN initiated with config: _collectorId:IOYL3BZ PAN_URL:https://XX.XX.XX.XX:443 PAN_USER:api PAN_PWD:******* nickName: notes: dpId: ENCRYPTED_CONFIG:true
 
Successful submit logs for PAN in collector:

INFO [2017-11-01 17:40:49,215] [U:349,F:091,T:440,M:2,121] core.common.DataProviderFactory:[DataProviderFactory:createNewDataProvider:131] - [Thread-1] - Creating data provider PAN_XX.XX.XX.XX for customer 11740 with config: _collectorId:IOYL3BZ nickName:pan8 notes: dpId: lastModifiedTimestamp:1509558049186 ENCRYPTED_CONFIG:true lastActivityTimestamp:1509558049198 dpState:ACTIVE PAN_URL:https://XX.XX.XX.XX:443 PAN_USER:api PAN_PWD:******* INFO [2017-11-01 17:40:49,215] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:74] - [Thread-1] - Data provider initialization started. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:79] - [Thread-1] - Config Validation succeeded. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:84] - [Thread-1] - DP operations manager successfully initiated. INFO [2017-11-01 17:40:49,216] [U:349,F:091,T:440,M:2,121] 197.17.11}:[Tasker:runTask:429] - [Thread-1] - Running 25: com.vnera.dataproviders.core.common.impl.dataprovider.AbstractDataProvider$InitializationAsyncTask_1 INFO [2017-11-01 17:40:49,229] [U:349,F:091,T:440,M:2,121] 197.17.11}:[Tasker$TaskerCallBack:<init>:111] - [Thread-1] - Creating callback 25: TaskContext {taskType=ASYNC, taskIdentifier=com.vnera.dataproviders.core.common.impl.dataprovider.AbstractDataProvider$InitializationAsyncTask_1} INFO [2017-11-01 17:40:49,230] [U:350,F:090,T:440,M:2,121] 197.17.11}:[AbstractDataProvider:initialize:93] - [Thread-1] - DP initialization done. WARN [2017-11-01 17:40:49,230] [U:350,F:090,T:440,M:2,121] core.dataprovidermanager.DataProviderManager:[DataProviderManager:startDataProvider:404] - [Thread-1] - INFO: DP instance started: PAN_1XX.XX.XX.XX for customerId XXXXX
 
  • CURL request to verify user creds validation:
curl -k -X GET "https://XX.XX.XX.XX/api/?type=keygen&user=api&password=admin"

Successful response:

<response status = 'success'><result><key>LUFRPT04MWw3ZzFFWFdpWGxqSElDQUZ5bUhMMVFIWWs9VVh1bURheGJTZXluRzJCS3U0dGNodz09</key></result></response>

 
If Validate Credentials still fails, make the following request from terminal (from proxy):

curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=<API_KEY>&cmd=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>" curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=LUFRPT04MWw3ZzFFWFdpWGxqSElDQUZ5bUhMMVFIWWs9VVh1bURheGJTZXluRzJCS3U0dGNodz09&cmd=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>"

Successful response:

<response status="success"><result><sync_status nsx_mgr_id="15"><last_dynamic_update>12:38AM Oct 05 2017</last_dynamic_update><status></status><sync_info>Registered</sync_info></sync_status> </result></response>

Error response:

curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=LUFRPT14UUpteExNaUhHd2FxQnZzdDV5NnJrQjZIUXM9RlRhTXV2ZkswWlk5MHprVG1YQ3lxdz09&cmd=<request><plugins><vmware_nsx><status></status></vmware_nsx></plugins></request>" <response status = 'error' code = '403'><result><msg>User not authorized to perform this operation.</msg></result></response>

 
For PAN7 make the following request:

curl -k -X GET "https://XX.XX.XX.XX/api/?type=op&action=show&key=LUFRPT04MWw3ZzFFWFdpWGxqSElDQUZ5bUhMMVFIWWs9VVh1bURheGJTZXluRzJCS3U0dGNodz09&cmd=?type=op&action=show&key=$ {API_KEY} &cmd=<request><partner><vmware-service-manager><status></status></vmware-service-manager></partner></request>" In case of error, cross check user created in panorama has valid XML API access.
 
Verifying PAN Data fetch from collector:

grep -nr "PANUtils" . ./collector.STDOUT-2017-10-31-22.06.42.log.error:392692:INFO [2017-11-01 11:32:56,565] [U:319,F:151,T:470,M:2,121] pan.utils.PANUtils:[PANUtils:init:145] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Init started ./collector.STDOUT-2017-10-31-22.06.42.log.error:392709:INFO [2017-11-01 11:32:58,857] [U:243,F:258,T:500,M:2,121] pan.utils.PANUtils:[PANUtils:init:166] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Init completed grep -nr "PANUtils" . | grep "Exception" grep -nr "PANUtils" . | grep "ERROR"

For Devices related parsing and fetch:

grep -nr "PanDevicesCommandParser" . collector.STDOUT-2017-11-01-06.57.08.log:178805:INFO [2017-11-01 09:45:43,335] [U:337,F:077,T:414,M:2,121] pan.parsers.PanDevicesCommandParser:[PANDeviceGroupCommandParser:getDeviceDetails:29] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - No devices found with deviceGroup name shared collector.STDOUT-2017-11-01-06.57.08.log:185571:ERROR [2017-11-01 09:50:43,610] [U:172,F:270,T:442,M:2,121] pan.parsers.PanDevicesCommandParser:[PanPhysicalHelper:parseVirtualSystemsAndZones:304] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - Failed to parse VirtualSystems And Zones collector.STDOUT-2017-11-01-06.57.08.log:185592:INFO [2017-11-01 09:50:43,610] [U:172,F:270,T:442,M:2,121] pan.parsers.PanDevicesCommandParser grep -nr "PanDevicesCommandParser" . | grep "Exception" grep -nr "PanDevicesCommandParser" . | grep "ERROR"
 
 
Verifying Platform Samza logs for exception around various PAN related entities:

sudo su cd /var/log/hadoop-yarn/containers/application_<>/ --------------------------------------------------------------------------------------- grep -nr "PANFirewallStoreConfigProgram" . | grep "Exception" grep -nr "PANServiceStoreConfigProgram" . | grep "Exception" grep -nr "PANManagerStoreConfigProgram" . | grep "Exception" grep -nr "PANDeviceGroupStoreConfigProgram" . | grep "Exception" grep -nr "PANDeviceStoreConfigProgram" . | grep "Exception" grep -nr "ANLogicalInterfaceStoreConfigProgram". | grep "Exception" grep -nr "PANVirtualRouterStoreConfigProgram" . | grep "Exception" grep -nr "PANVirtualSystemStoreConfigProgram" . | grep "Exception" grep -nr "PANZoneStoreConfigProgram". | grep "Exception"

To validate if platform received specific pan message type sdm, do following:

Collecting SDM dumps for engineering analysis

Validate in vRNI proxy, if Collector is sending message to Platform:

grep -nr "sending message key:" | grep "com.vnera.model.pan.Addressbase.config" collector.STDOUT-2017-10-31-22.06.42.log.error:263270:INFO [2017-11-01 07:24:42,903] [U:277,F:179,T:456,M:2,121] dataprovider.utils.CollectorUtils$PayloadCache:[CollectorUtils:pushPayLoadToCollector:281] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - sending message key:hashKey:https://XX.XX.XX.XX:443, upstreamDataType=com.vnera.model.pan.Addressbase.config, consumerIdentifier=1, dpIdentifier=11740_PAN_XX.XX.XX.XX payloadHash:f224ef4d49d3eb1fab2e6be7177164f01ce2cb1e size:7779 type:com.vnera.model.pan.Addressbase.config hash:https://XX.XX.XX.XX:443 collector.STDOUT-2017-10-31-22.06.42.log.error:392710:INFO [2017-11-01 11:32:58,985] [U:342,F:158,T:500,M:2,121] dataprovider.utils.CollectorUtils$PayloadCache:[CollectorUtils:pushPayLoadToCollector:281] - [Schedule_PAN_XX.XX.XX.XX_Inventory_OpMgr-0] - sending message key:hashKey:https://XX.XX.XX.XX:443, upstreamDataType=com.vnera.model.pan.Addressbase.config, consumerIdentifier=1, dpIdentifier=11740_PAN_XX.XX.XX.XX payloadHash:f224ef4d49d3eb1fab2e6be7177164f01ce2cb1e size:7779 type:com.vnera.model.pan.Addressbase.config hash:https://XX.XX.XX.XX:443
 
Then on Platform do following:

create folder to collect dump:

mkdir -p /home/ubuntu/sdm-dump/


Run following KafkaReader command to dump sdms of specific message type, replace value of messageType param accordingly:

java -cp ~/build-target/common-utils/tools-0.001-SNAPSHOT.jar com.vnera.tools.KafkaReader 0 Topic3 -1 localhost 9092 0 -messageType com.vnera.model.pan.Addressbase.config -dumpSDMsToFile /home/ubuntu/sdm-dump/pan

Various PAN message types
--------------------------------------
com.vnera.model.pan.DeviceGroup.config
com.vnera.model.pan.Addressbase.config
com.vnera.model.pan.Servicebase.config
com.vnera.model.pan.Firewall.config
com.vnera.model.pan.PANManager.config
com.vnera.model.pan.PhysicalDevice.config
com.vnera.model.pan.NSXVMSeriesDevice.config
com.vnera.model.pan.LogicalDevice.config
com.vnera.model.pan.VirtualRouter.config
com.vnera.model.pan.VirtualSystem.config
com.vnera.model.pan.Zone.config
com.vnera.model.pan.LogicalInterface.config