Database VM experiencing network latency

Products

• VMware vSphere ESXi

Issue/Introduction

• Communication between VMs seeing considerable spikes of latency (for example, between OpenShift WORKER VM and Oracle DB VM).
  
  • Spikes as high as 100MS were recorded making the application unusable. 
• No latency seen between VM's on same L2 (even across hosts).
• Usually the slowness is reported during a very specific time frame (for example, in the morning between 7am and 1pm. This time may vary depending on environment).
  
  • Backups are known to run during this time.
• Using traceroute from between the problematic VMs  continuously shows intermittent latency spikes between specific hops (for example, between "worker" VM to Oracle DB VM at hop 4 & 5).
  

  traceroute to <Oracle DB VM FQDN> (#.#.#.#), 30 hops max, 60 byte packets
  1  * * *
  2  #.#.#.# (#.#.#.#)  1.231 ms  1.304 ms  1.174 ms
  3  #.#.#.# (#.#.#.#)  0.952 ms #.#.#.# (#.#.#.#)  1.348 ms  1.334 ms
  4  #.#.#.# (#.#.#.#)  18.511 ms  18.480 ms  0.848 ms
  5  #.#.#.# (#.#.#.#)  18.501 ms  0.915 ms  1.377 ms
  6  #.#.#.# (#.#.#.#)  1.086 ms *  1.177 ms
  
  

  
  
  • Sometimes latency is seen only on 1 hop.
    
    

Environment

VMware vSphere ESXi

 

Cause

• Backup process is consuming excessive bandwidth between the hours of 7am and 1pm EST. 
  • NOTE: This time may vary depending on enviornment.

Resolution

• Stop backup process and/or spread backups across a longer time-frame.

Additional Information

Troubleshooting ESX/ESXi virtual machine performance issues