ESXi host loses connectivity to storage and is unable to reconnect with EMC VPLEX
search cancel

ESXi host loses connectivity to storage and is unable to reconnect with EMC VPLEX

book

Article ID: 310823

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • ESXi host loses connection to EMC VPLEX.
  • The host is not able to reconnect until it is rebooted.
  • In the vmkernel logs, you see this sequence of events:

    All the remote ports are down
    vmkernel.1:2012-11-05T23:17:27.114Z cpu11:2690)<3> rport-8:0-0: blocked FC remote port time out: saving binding
    vmkernel.1:2012-11-05T23:17:27.983Z cpu4:2693)<3> rport-7:0-1: blocked FC remote port time out: saving binding
    vmkernel.1:2012-11-05T23:17:34.991Z cpu10:2682)<3> rport-7:0-0: blocked FC remote port time out: saving binding
    vmkernel.1:2012-11-05T23:17:35.122Z cpu5:2687)<3> rport-8:0-1: blocked FC remote port time out: saving binding
  • If you are using EMC PowerPath:
    • It reports the ports as dead, then alive again:

      2012-11-05T23:17:30.418Z cpu0:2705)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Bus 1 to Invista FNM00105200194 port 00 is dead.
      2012-11-05T23:17:49.741Z cpu9:2701)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Bus 2 to Invista FNM00105200194 port 09 is dead.
      2012-11-05T23:17:49.752Z cpu8:2701)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Bus 2 to Invista FNM00105200194 port 01 is dead.
      2012-11-05T23:17:49.769Z cpu12:2701)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Bus 1 to Invista FNM00105200194 port 08 is dead.

      2012-11-05T23:22:21.283Z cpu12:2716)PowerPath: EmcpEsxLogEvent:1252: Info:Mpx:Bus 1 to Invista FNM00105200194 port 08 is alive.
      2012-11-05T23:22:21.641Z cpu13:2716)PowerPath: EmcpEsxLogEvent:1252: Info:Mpx:Bus 1 to Invista FNM00105200194 port 00 is alive.
      2012-11-05T23:22:21.970Z cpu14:2705)PowerPath: EmcpEsxLogEvent:1252: Info:Mpx:Bus 2 to Invista FNM00105200194 port 01 is alive.
      2012-11-05T23:22:22.457Z cpu11:2716)PowerPath: EmcpEsxLogEvent:1252: Info:Mpx:Bus 2 to Invista FNM00105200194 port 09 is alive.
    • The paths to the storage are marked as dead:

      2012-11-05T23:17:30.418Z cpu0:2705)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Path vmhba8:C0:T0:L28 to FNM00105200194 is dead.
      2012-11-05T23:17:30.418Z cpu0:2705)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Path vmhba8:C0:T0:L27 to FNM00105200194 is dead.
      2012-11-05T23:17:30.418Z cpu0:2705)ALERT: PowerPath: EmcpEsxLogEvent:1260:Error:Mpx:Path vmhba8:C0:T0:L29 to FNM00105200194 is dead.
    • Commands are failing with a Check Condition from the array with Sense Key = NOT READY, Additional Sense Code/Qualifier = LOGICAL UNIT NOT READY, MANUAL INTERVENTION REQUIRED:

      2012-11-05T23:18:00.509Z cpu9:2705)ScsiDeviceIO: 2322: Cmd(0x4124403c26c0) 0x2a, CmdSN 0x80000007 from world 286368 to dev "naa.6000144000000010a00f545a97c570bd" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.
      2012-11-05T23:18:00.509Z cpu9:2705)ScsiDeviceIO: 2322: Cmd(0x4124010010c0) 0x2a, CmdSN 0x80000033 from world 900201 to dev "naa.6000144000000010a00f545a97c5702c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.

  • Similar messages display for Native Multipathing Plugin.


Environment

VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable

Cause

The VPLEX returns these SCSI codes when the connection between the primary and the secondary is lost. The issue is related to an internal timer unit mismatch of the watchdogs in the DMA driver on the VPLEX.

This is not a VMware issue.

Resolution

The issue is resolved in the VPLEX patch set 5.1 Patch 3. Contact EMC for more information.

Powered by Wolken Software