pod logs showing “i/o timeout” intermittently in TKGs Guest Cluster

Products

• VMware Tanzu Kubernetes Grid Service (TKGs)

Issue/Introduction

• Similar logs can be seen from pod log.
  
  kubectl logs <pod name>
  [ERROR]        FOLLOWER <pod name> send.ping.to.peer <pod name> new.client.error[dial tcp: i/o timeout]
• Pod availability might be impacted.

Environment

VMware Tanzu Kubernetes Grid Service (TKGs)

Cause

Those I/O timeout error usually indicating networking interruptions.

Resolution

Isolate the networking issue in following sequence:

1. Identify the networking failure by pod to pod packet capture.
   Note not only examining the pod production traffic but also checking DNS reolutions to coredns.
2. Packet capture at worker node level.
   If the packet loss is not observed within worker node, then underlaying networking should be checked further.