Traffic encapsulated in-guest using VxLAN will be dropped if a non-default VxLAN port is used while VxLAN hardware offloading is enabled on the vmxnet3 adapter.
This is a known behavior within ESXi.
The issue is fixed in ESXi 7.0.3 P09 release name "ESXi 7.0 Update 3q" build 23794027.
Refer kb https://knowledge.broadcom.com/external/article/372545/ to download the patch from Broadcom portal.