SR-IOV: VF "Link Down" is not detected when PF is down for Mellanox ConnectX-5 and later generation card's

Products

• VMware vSphere ESXi

Issue/Introduction

For Mellanox ConnectX-4 , link states of VF (network cards in guest OS) follows link state of PF (ConnectX-4). That means we will see such link state change events within guest OS when link state of ConnectX-4 changes in ESXi,
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [   17.748038] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [   17.943809] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [26590.829883] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [26590.829911] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [26600.730429] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [26600.730468] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [31072.355949] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [31072.355951] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [31087.405213] mlx5_core 0000:xx:zz.0 ens67: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [31087.405261] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [100389.859391] mlx5_core 0000:xx:yy.0 ens66: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [100389.859410] mlx5_core 0000:xx:zz.0 ens67: Link down
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [100919.157808] mlx5_core 0000:xx:yy.0 ens66: Link up
syslog:[YYYY-MM-DDTHH:MM:SS]  passthru kernel: [100919.157865] mlx5_core 0000:xx:zz.0 ens67: Link up

But for Mellanox ConnectX-5 and later generation card's, link states of VF does not follow link state of PF (For Mellanox ConnectX-5  and later generation card's,  if the link state changes on the Physical Port/PF from up to down or vice versa, we do not see any state change for VF's state)
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [   17.471951] mlx5_core 0000:xx:00.0 ens64: Link up
syslog:[YYYY-MM-DDTHH:MM:SS] passthru kernel: [   17.620611] mlx5_core 0000:xx:01.0 ens65: Link up

Cause

For older nic firmware, VF follows PF state. However in newer firmware VF does not follow PF state. This is by design.

Mellanox ConnectX-4 uses a very old firmware and has no new firmware to update, so the behavior is different than ConnectX-5 and later generation card's with newer firmware.

Resolution

Nvidia updated that driver does not have any workaround for VF to follow PF state.