"warning: more than 1 hinted handoff file detected","partition":"/var/vcap/store/router/handoff/##/<LOCAL_VM_IP_ADDRESS>:9000","files":"23,24"
Example showing folder 20 and files 23 and 24 (using fake IP 10.10.1.50 as local VM IP address):"warning: more than 1 hinted handoff file detected","partition":"/var/vcap/store/router/handoff/20/10.10.1.50:9000","files":"23,24"
This problem is not version specific, it was seen on Metric-Store 1.7.0.
The underlying storage outage leads to corruption on the log-store VM's filesystem. Some handoff files are corrupted and cause the router service to fail when attempting to process them.
Move the offending files to a backup folder created under /var/vcap/store then restart the router service:
# mkdir /var/vcap/store/log_bak
# mv /var/vcap/store/router/handoff/##/<LOCAL_VM_IP_ADDRESS>\:9000/23 /var/vcap/store/log_bak
# mv /var/vcap/store/router/handoff/##/<LOCAL_VM_IP_ADDRESS>\:9000/24 /var/vcap/store/log_bak
# mv /var/vcap/store/router/handoff/20/10.10.1.50\:9000/23 /var/vcap/store/log_bak
# sudo monit restart router
Note that this will restore the router service, however, the logging present in files 23 and 24 will not be persisted as the files are corrupted and unrecoverable.