Error messages seen in the Telegraf pod logs including "error making HTTP request" and "Unable to gather disk name"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The VKS telegraf logs are listing HTTP request connection refused error messages to 127.0.0.1 and to the IP addresses of the worker nodes.  Entries may look similar to:

  • Error in plugin: error making HTTP request to "https://127.0.0.1:6443/metrics": Get "https://127.0.0.1:6443/metrics": dial tcp 127.0.0.1:6443: connect: connection refused

• VKS telegraf logs list entries stating "Unable to gather disk name".  Entries may look similar to:

  • stderr F 2026-##-##T##:##:##Z W! [inputs.diskio::host_diskio] Unable to gather disk name for "sda": error reading /dev/sda: no such file or directory

Environment

• VCF 9.x
• VKS 9.x

Cause

• The default configuration of the current telegraf package deploys the telegraf daemonset pod to all of the nodes (both control plane and worker nodes).  The worker nodes however, do not have Kubernetes components such as etcd server and kube-apiserver.  The lack of these components causes the HTTP errors that are seen in the telegraf logs of the worker nodes.  This is expected behavior.
  
  
• The "Unable to gather disk name" messages are caused by telegraf attempting to collect all of the disk devices on the node, including /dev/sda, /dev/sda1, etc, but telegraf is currently unable to access the full dev devices.  Telegraf is still able to collect diskio metrics, just not from the dev device path.  This is also expected behavior.  

Resolution

• The HTTP connection refused messages in the telegraf logs of the worker nodes can be ignored.
  
  
• The "Unable to gather disk name" messages can also be ignored. 

Additional Information

Japanese version: Telegraf ポッドのログに表示されるエラーメッセージには、「HTTP リクエストの実行エラー」や「ディスク名を取得できません」などがあります。