SR-IOV: VF "Link Down" is not detected when PF is down for Mellanox ConnectX-5

Products

• VMware vSphere ESXi

Issue/Introduction

For Mellanox ConnectX-4 , link states of VF (network cards in guest OS) follows link state of PF (ConnectX-4). That means we will see such link state change events within guest OS when link state of ConnectX-4 changes in ESXi,
syslog:May 30 10:12:38 passthru kernel: [   17.748038] mlx5_core 0000:03:02.0 ens66: Link up
syslog:May 30 10:12:38 passthru kernel: [   17.943809] mlx5_core 0000:03:03.0 ens67: Link up
syslog:May 30 17:35:31 passthru kernel: [26590.829883] mlx5_core 0000:03:03.0 ens67: Link down
syslog:May 30 17:35:31 passthru kernel: [26590.829911] mlx5_core 0000:03:02.0 ens66: Link down
syslog:May 30 17:35:41 passthru kernel: [26600.730429] mlx5_core 0000:03:03.0 ens67: Link up
syslog:May 30 17:35:41 passthru kernel: [26600.730468] mlx5_core 0000:03:02.0 ens66: Link up
syslog:May 30 18:50:13 passthru kernel: [31072.355949] mlx5_core 0000:03:02.0 ens66: Link down
syslog:May 30 18:50:13 passthru kernel: [31072.355951] mlx5_core 0000:03:03.0 ens67: Link down
syslog:May 30 18:50:28 passthru kernel: [31087.405213] mlx5_core 0000:03:03.0 ens67: Link up
syslog:May 30 18:50:28 passthru kernel: [31087.405261] mlx5_core 0000:03:02.0 ens66: Link up
syslog:May 31 14:05:30 passthru kernel: [100389.859391] mlx5_core 0000:03:02.0 ens66: Link down
syslog:May 31 14:05:30 passthru kernel: [100389.859410] mlx5_core 0000:03:03.0 ens67: Link down
syslog:May 31 14:14:20 passthru kernel: [100919.157808] mlx5_core 0000:03:02.0 ens66: Link up
syslog:May 31 14:14:20 passthru kernel: [100919.157865] mlx5_core 0000:03:03.0 ens67: Link up

But for Mellanox ConnectX-5, link states of VF does not follow link state of PF (ConnectX-50. No matter link state of ConnectX-5 is down or up, we do not see any state change for VF,
syslog:May 30 10:12:38 passthru kernel: [   17.471951] mlx5_core 0000:03:00.0 ens64: Link up
syslog:May 30 10:12:38 passthru kernel: [   17.620611] mlx5_core 0000:03:01.0 ens65: Link up

Cause

For older nic firmware, VF follows PF state. However in newer firmware VF does not follow PF state. This is by design.

Mellanox ConnectX-4 uses a very old firmware and has no new firmware to update, so the behavior is different than ConnectX-5 with new firmware.

Resolution

Nvidia updated that driver does not have any workaround for VF to follow PF state.