AMD MILAN 2P and 100G Networking Best Practices
Article ID: 315538
Updated On:
Products
Issue/Introduction
Overview:
This article provides best practices for improving stability and reliability when dealing with high network performance requirements.
Symptoms:
In some cases, a 100G network adapter may exhibit reduced performance during periods of high packet flow and throughput.
Best Practice:
The recommended best practice applies specifically to the AMD Milan 2P platform. Single processor (1P) systems are not affected by this issue.
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
Resolution
PCI Preferred setting only applies to one singular PCI-Bus ID by prioritizing the access to the PCI entity. Its not only limited to network cards and any specified type might benefit from the change.
If dual/quad port network cards are used, all network port on the PCI card might benefit from the change.
Note: Only one PCI-ID can be specified.
Identify PCI-ID on ESXi Host
Example 1:
esxcfg-nics -l
or
esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ---------- ------------ ----------- ------ ------ ----------------- ---- -----------
vmnic0 0000:e1:00.0 ntg3 Up Down 0 Half ##:##:##:##:##:## 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1 0000:e1:00.1 ntg3 Up Down 0 Half ##:##:##:##:##:## 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2 0000:63:00.0 nmlx5_core Up Up 25000 Full ##:##:##:##:##:## 1500 Mellanox Technologies MT27800 Family [ConnectX-5]
vmnic3 0000:63:00.1 nmlx5_core Up Up 25000 Full ##:##:##:##:##:## 1500 Mellanox Technologies MT27800 Family [ConnectX-5]
vmnic4 0000:a1:00.0 nmlx5_core Up Up 100000 Full ##:##:##:##:##:## 1500 Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
vmnic5 0000:a1:00.1 nmlx5_core Up Up 100000 Full ##:##:##:##:##:## 1500 Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
In the above example, we can find 3x duo network cards, PCI-ID HEX: 0Xe1, 0x63, and 0xa1.
Example 2: Identify PCI-BUS ID through hardware device, vmnic5:
vmkchdev -l|grep vmnic5
vmkchdev -l|grep vmnic5
0000:a1:00.1 15b3:101d 15b3:0058 vmkernel vmnic5
PCI-ID: 0000:a1:00.1
Hardware Vendor ID: #### #### #### ####
Example 3: PowerCLI
Using get-esxcli commandlet:
$esxcli = Get-EsxCli -VMHost <hostip/fqdn> -V2
$esxcli.network.nic.list.Invoke()|ft Name,PCIDevice
output - example:
Name PCIDevice
---- ---------
vmnic0 0000:e1:00.0
vmnic1 0000:e1:00.1
vmnic2 0000:63:00.0
vmnic3 0000:63:00.1
vmnic4 0000:a1:00.0
vmnic5 0000:a1:00.1
Values are in HEX and require translation to decimal for the Host BIOS.
Example: ESXi shell to translate HEX to DEC:
printf "%d\n" 0xa1
161
PCI-ID "161" to be set in the Host BIOS
Add changes in Dell Bios as example:
- Configuration -> BIOS Settings -> Integrated Devices:
BIOS changes requires host boot to set the applied changes active.
Additional Information
Additional Best Practice Guidance:
vSphere Performance Best Practice Guidance: https://www.vmware.com/docs/vsphere-esxi-vcenter-server-70u3-performance-best-practices
AMD: https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/amd-epyc-7002-tg-hpc-56827.pdf
Dell: https://dl.dell.com/content/manual25828732-dell-emc-poweredge-r7525-bios-and-uefi-reference-guide.pdf?language=en-us&ps=true
Lenovo: https://lenovopress.lenovo.com/lp1267.pdf
HPe: https://support.hpe.com/hpesc/public/docDisplay?docId=a00112581en_us&docLocale=en_US&page=GUID-9697C877-2BA3-4DB1-B643-A20FAF3A9144.html
vSphere 7.0U2+ - AMD Performance optimization: https://blogs.vmware.com/performance/2021/03/vsphere70u2-cpu-scheduler-amd-epyc-processors-performance.html
Impact/Risks:
Applicable to AMD Milan 2P platform.
Feedback
