VMware vSphere ESXi
This is not an ESXi network performance issue.
SMB protocol performance is not only limited to network speed, but also SMB flow control mechanism.
Refer to the following documentation for SMB performance tuning.
Performance tuning for SMB file servers
https://learn.microsoft.com/en-us/windows-server/administration/performance-tuning/role/file-server/smb-file-server
Slow SMB files transfer speed
https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/slow-smb-file-transfer