Missing Application attributes on Splunk logs

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
When you check on logs received from Splunk nozzle apps, some of the log entries are incomplete with missing application attributes. These attributes might be one or more i.e , cf_org_id, cf_org_name etc.

Refer to the sample log below:

Notice the log with time 2/25/19 2:13:34.851 PM and the cf_app_name, cf_org_name, cf_org_id are some of the attributes that are missing.

2/25/19 2:13:34.883 PM {[-]
	cf_app_id: 643edd33-2bc1-4f84-bd81-1-1d80fa055be9
         cf_app_name: my-app-name
         cf_org_id: 49fa3322-84fe-4157-93d2-a29a3819fff6
         cf_org_name: my_org_name
         cf_origin: firehose
         cf_space_id: 39a93b b3b-3276-4658-g5ba11dde09a
         cf_space_name: my_space_name
         deployment:cf
         event_type: LogMessage
         ip: 10.123.12.12
         job: router
         job_index: f8623a-8c1c-43a2-aaa8-85ca5af980123
         message_type: OUT
         msg : <msg>
         origin: go_router
         source_instance:2
         source_type: RTR
         timestamp: 1551122014882610770
         }


2/25/19 2:13:34.851 PM {[-]
	cf_app_id: 643edd33-2bc1-4f84-bd81-1-1d80fa055be9
         cf_origin: firehose
         deployment:cf
         event_type: LogMessage
         ip: 10.123.12.12
         job: router
         job_index: f8623a-8c1c-43a2-aaa8-85ca5af980123
         message_type: OUT
         msg : <msg>
         origin: go_router
         source_instance:2
         source_type: RTR
         timestamp: 1551122014882610800
         }

Environment

Cause

Missing attributes are attributed with:

Attributes have nil value when TAS pass on to Splunk
Splunk nozzle apps are unable to show those attributes.

Resolution

Here are some troubleshooting steps to resolve this issue:

Verify using the curl cmd to get app attributes from Cloud controller api. Check if the attributes are missing or have nil values.
```
cf curl /v2/apps?inline-relations-depth=2&order-direction=desc&results-per-page=100
```
If this is the case, please open a VMware support ticket and attach the results of the command above plus your cloud controller logs. You might be asked for more logs when a Support Engineer starts working on the ticket.
If you don't see anything missing on the command from step 1, check your Splunk nozzle app to figure out why it's unable to display those attributes. In most support cases, restarting the Splunk nozzle app will resolve this issue. If this does not help, please open a ticket with Splunk Support as the nozzle itself is third party software and not supported by VMware.