log-store vm failing with tsdb error index out of range in App Metrics 2.2.2

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

• Log-store VM keep getting failing, due to log-store process issue.

log-store-vms/vs0f2109-f065-xxxx-xxxx-8de3bhahd1153:~$ sudo monit summary
The Monit daemon 5.2.5 uptime: 9m 

Process 'cf-auth-proxy'             running
Process 'log-store'                 Does not exist
Process 'loggregator_agent'         running
Process 'nozzle'                    running
Process 'router'                    running
Process 'prom_scraper'              running
Process 'route_registrar'           running
Process 'system-metrics-agent'      running
Process 'minio-dns-alias'           running
Process 'bosh-dns'                  running
Process 'bosh-dns-resolvconf'       running
Process 'bosh-dns-healthcheck'      running
System 'system_vs0f2109-f065-xxxx-xxxx-8de3bhahd1153' running

• log-store.stderr.log error message. 

{"level":"info","timestamp":"2025-03-24T13:02:34.405025216Z","caller":"logger/fields.go:110","message":"Open store (end)","service":"store","trace_id":"0vU3oC1G000","op_name":"tsdb_open","op_event":"end","op_elapsed":"927.041185ms"}{"level":"info","timestamp":"2025-03-24T13:02:34.40517083Z","caller":"tsdb/store.go:214","message":"Using data dir","service":"store","path":"/var/vcap/store/log-store/influxdb/21/data"}{"level":"info","timestamp":"2025-03-24T13:02:34.405194364Z","caller":"tsdb/store.go:287","message":"Compaction settings","service":"store","max_concurrent_compactions":4,"throughput_bytes_per_second":50331648,"throughput_bytes_per_second_burst":50331648}{"level":"info","timestamp":"2025-03-24T13:02:34.405208518Z","caller":"logger/fields.go:108","message":"Open store (start)","service":"store","trace_id":"0vU3oFeG000","op_name":"tsdb_open","op_event":"start"}panic: runtime error: index out of range [1] with length 0 

goroutine 109 [running]:

encoding/binary.bigEndian.Uint16(...)        /var/vcap/data/packages/golang-1.21-linux/021e45575b86d05c243cb3069d8b453768456296/src/encoding/binary/binary.go:139github.com/influxdata/influxdb/tsdb.ReadSeriesKeyMeasurement(...)        /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/series_file.go:380github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).execSeriesEntry(0xc021c78000, 0xc022227c88)

 

Environment

Product Version: 2.x

Cause

• From the log observation it seems to be index files(tsl/tsi) got corrupted. Also, Shard 21 got corrupted while it is getting restarted.

{"level":"info","timestamp":"2025-03-24T12:47:46.296241546Z","caller":"tsdb/store.go:214","message":"Using data dir","service":"store","path":"/var/vcap/store/log-store/influxdb/21/data"}{"level":"info","timestamp":"2025-03-24T12:47:46.296267982Z","caller":"tsdb/store.go:287","message":"Compaction settings","service":"store","max_concurrent_compactions":4,"throughput_bytes_per_second":50331648,"throughput_bytes_per_second_burst":50331648}{"level":"info","timestamp":"2025-03-24T12:47:46.296280169Z","caller":"logger/fields.go:108","message":"Open store (start)","service":"store","trace_id":"0vU2y2U0000","op_name":"tsdb_open","op_event":"start"}panic: runtime error: index out of range [1] with length 0
goroutine 97 [running]:encoding/binary.bigEndian.Uint16(...)	/var/vcap/data/packages/golang-1.21-linux/021e45575b86d05c243cb3069d8b453768456296/src/encoding/binary/binary.go:139 

Resolution

• In order to fix this there is a workaround which you can follow to remove index files and restart log-store VM. 
  
  

1. ssh to the log-store vm and move to root using sudo -i
   

   bosh -d appMetrics-ab42f2c3f83c6874ce83 ssh log-store-vms/vs0f2109-f065-xxxx-xxxx-8de3bhahd1153

   
   
   
2. monit stop log-store
3. mkdir -p /var/vcap/store/log-store/influxdb/backups
4. cp -r /var/vcap/store/log-store/influxdb/21 /var/vcap/store/log-store/influxdb/backups/21
5. find /var/vcap/store/log-store/influxdb/21/data -type f \( -name "*.tsi" -o -name "*.tsl" \) -exec rm -v {} \;
6. rm -rf /var/vcap/store/log-store/influxdb/21/data/index
7. monit start log-store
   
   
   Note** Once log-store is started index data should be auto generated again and log-store vm should be in running state.

Additional Information

• You can validate if tsm files are healthy by running the below query. Also, you can compare the output from a healthy log-store vm as well.

for i in /var/vcap/store/log-store/influxdb/*; do   echo "Processing: $i";   sudo /var/vcap/packages/influx-inspect/influx_inspect verify -dir "$i"; done