Meraki Plugin Fails to Send Inventory to Data Collector in DX NetOps

Products

• Network Observability
• CA Performance Management

Issue/Introduction

The UCF is configured with the Meraki plugin, but no inventory flows from the UCF to the Data Collector and Performance Manager ․​​​​​​​‍

 

ERROR MESSAGE: "no handler for route 'secret/config'․ route entry not found․"

 

SYMPTOMS:

• Plugin configured but no inventory data received

• The [meraki-poller-pod] pod is idle and fails to execute polling cycles

• Meraki plugin fails to save credentials to the Vault with permission denied and invalid token errors

• 403 errors returned when testing Vault rootkey

   

CONTEXT: Occurs after initial deployment of the Meraki plugin in a Kubernetes environment ․

 

IMPACT: Inventory and event data from Meraki devices are not monitored or collected ․

Environment

• OS: Kubernetes

• Database: Postgres

• App Server: DX NetOps Cloud-Native VNA (UCF) 25․4․4

• Plugin: Meraki Plugin

Resolution

1. Checking Vault Database Credentials

To verify if the token stored in the Postgres database is recognized by Vault, use the following commands to extract the root key:

 

• Extract the root key from Postgres:

   

  kubectl exec -it postgres-0 -n platform -- bash -c "PGPASSWORD=\$(kubectl get secret ucf-pguser-secret -n netops -o json | jq -r .data.password | base64 -d) psql -d keydb -U \$(kubectl get secret ucf-pguser-secret -n netops -o json | jq -r .data.username | base64 -d)"
  

   

  (Note: After accessing the database, run the SQL query below)

   

• SQL Query:

  
  

   

  select rootkey from unseal;
  

2. Testing Vault Access & Health

Once you have the root key, you can check Vault's status and test the route:

 

• Check Vault's current health and sealed/initialized state:

   

  kubectl exec vault-0 -n platform -- vault status
  

• Test the secret configuration route:

   

  curl --header "X-Vault-Token: <rootkey>" http://vault.platform.svc.cluster.local:8200/v1/secret/config
  

• If you need to run the curl command from within a specific pod:

   

  kubectl exec -it <meraki-admin-pod-name> -n netops -- /bin/sh
  

3. Fixing the Vault Mount Path (403 Errors)

If the Vault initialized correctly but the KV-v2 secret engine is not mounted at the /secret path, mount it manually:

 

• Mount the secret engine:

   

  kubectl exec vault-0 -n platform -- sh -c 'vault login <rootkey> > /dev/null && vault secrets enable -path=secret -version=2 kv'
  

4. Resolving Kafka Partition Loss

To investigate if the vna-polling-scheduler consumer group lost its partition assignments due to connection issues, describe the consumer group:

 

• Check partition assignment and lag:

   

  kubectl exec -it kafka-cluster-kafka-cluster-pool-broker-0 -n platform -- /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group vna-polling-scheduler
  

5. Clearing Stale Data and Corrupted Kafka State

If the UI is showing stale Engine IDs or IP Domain IDs and the platform is experiencing CrashLoopBackoff errors due to an invalid cluster ID, perform a clean uninstall of the application layer:

 

• Clean the product installation:

   

  tcxctl clean product
  

6. Gathering Namespace Logs

If further investigation is required and old data cannot be deleted via the UI, a bash script is provided in the notes to gather logs from all pods in a namespace:

 

• Retrieve the list of pods:

   

  kubectl get pods -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}'
  

• Capture logs for all containers in a pod:

   

  kubectl logs -n "$NAMESPACE" "$pod" --all-containers=true > "$OUTPUT_FILE" 2>&1
  

• Compress the logs into an archive:

   

  tar -czvf "$ARCHIVE_NAME" *.log