The physical NIC is incorrectly dropping packets.
This is a known issue in ESXi 8.0 U3. Broadcom Engineering is aware of it and is actively working on a fix, which will be included in an upcoming ESXi release.
If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article for a workaround.
For more information, see Creating and managing Broadcom support cases.