When you initially start a download, the TCP windows are empty and the server sends data as fast as possible. On the receiving end, depending on your uplink speed, eventually the TCP buffers will start getting backed up and it will tell the server to stop sending too much data by reducing the TCP window size. This causes the bandwidth to drop and this resizing of the TCP window happens until the client and server reach a reasonable median where they are able to handle the traffic and the download rate becomes fairly steady.
Issue maybe resolved by disabling Large Receive Offload on a VC, however this is not possible in a VMC environment.
LRO reassembles incoming network packets into larger buffers and transfers the resulting larger but fewer packets to the network stack of the host or virtual machine. The CPU has to process fewer packets than when LRO is disabled, which reduces its utilization for networking