SR-IOV: VF "Link Down" is not detected when PF is down for Mellanox ConnectX-5 and later generation card's
Article ID: 372121
Updated On:
Products
Issue/Introduction
For Mellanox ConnectX-4 , link states of VF (network cards in guest OS) follows link state of PF (ConnectX-4). That means we will see such link state change events within guest OS when link state of ConnectX-4 changes in ESXi,
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [ 17.748038] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [ 17.943809] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [26590.829883] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [26590.829911] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [26600.730429] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [26600.730468] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [31072.355949] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [31072.355951] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [31087.405213] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [31087.405261] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [100389.859391] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [100389.859410] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [100919.157808] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [100919.157865] mlx5_core 0000:xx:zz.0 ens67: Link up
But for Mellanox ConnectX-5 and later generation card's, link states of VF does not follow link state of PF (For Mellanox ConnectX-5 and later generation card's, if the link state changes on the Physical Port/PF from up to down or vice versa, we do not see any state change for VF's state)
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [ 17.471951] mlx5_core 0000:xx:00.0 ens64: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [ 17.620611] mlx5_core 0000:xx:01.0 ens65: Link up
Environment
VMware ESXi 8.x
Cause
For older nic firmware, VF follows PF state. However, in newer firmware VF does not follow PF state. This is by design.
Mellanox ConnectX-4 uses a very old firmware and has no new firmware to update, so the behavior is different than ConnectX-5 and later generation card's with newer firmware.
Resolution
This issue is resolved with ESX 8.0 Update 3j, in the NVIDIA nmlx5 driver, you can configure the behavior of the VF link state. Download nmlxcli tools, which are esxcli command line extensions available on the NVIDIA website, on the ESXi host to configure PFs so that VFs follow the PF link state.
For further assistance with the nmlxcli tools and configuration, please reach out to NVIDIA.
Additional Information
Feedback
