Database VM experiencing network latency
search cancel

Database VM experiencing network latency

book

Article ID: 420580

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Communication between VMs seeing considerable spikes of latency (for example, between OpenShift WORKER VM and Oracle DB VM).
    • Spikes as high as 100MS were recorded making the application unusable. 
  • No latency seen between VM's on same L2 (even across hosts).
  • Usually the slowness is reported during a very specific time frame (for example, in the morning between 7am and 1pm. This time may vary depending on environment).
    • Backups are known to run during this time.
  • Using traceroute from between the problematic VMs  continuously shows intermittent latency spikes between specific hops (for example, between "worker" VM to Oracle DB VM at hop 4 & 5).
    traceroute to <Oracle DB VM FQDN> (#.#.#.#), 30 hops max, 60 byte packets
    1  * * *
    2  #.#.#.# (#.#.#.#)  1.231 ms  1.304 ms  1.174 ms
    3  #.#.#.# (#.#.#.#)  0.952 ms #.#.#.# (#.#.#.#)  1.348 ms  1.334 ms
    4  #.#.#.# (#.#.#.#)  18.511 ms  18.480 ms  0.848 ms
    5  #.#.#.# (#.#.#.#)  18.501 ms  0.915 ms  1.377 ms
    6  #.#.#.# (#.#.#.#)  1.086 ms *  1.177 ms
    
    

    • Sometimes latency is seen only on 1 hop.

Environment

VMware vSphere ESXi

 

Cause

  • Backup process is consuming excessive bandwidth between the hours of 7am and 1pm EST. 
    • NOTE: This time may vary depending on enviornment.

Resolution

  • Stop backup process and/or spread backups across a longer time-frame.

Additional Information

Troubleshooting ESX/ESXi virtual machine performance issues