vSAN iSCSI target connection fails from a different subnet or VLAN

Products

• VMware vSphere ESXi

Issue/Introduction

An iSCSI initiator located on a different IP subnet than the vSAN iSCSI network cannot discover or connect to the vSAN iSCSI target service. The connection attempt to TCP port 3260 times out.

The behavior is identical whether the connection uses the vSAN iSCSI virtual IP address or an individual ESXi host VMkernel address. The behavior is also identical regardless of which ESXi host in the vSAN cluster the initiator runs on.

An initiator placed directly on the vSAN iSCSI network and portgroup discovers the target and mounts the LUN without issue.

This occurs even when no firewall exists between the two networks, both subnets reside in the same virtual routing and forwarding (VRF) instance, and both switched virtual interfaces (SVIs) reside in that same VRF.

Confirming the issue matches

Run the following checks on the ESXi host that owns the vSAN iSCSI virtual IP address. Substitute the values that apply to the environment:

• <initiator-ip> is the IP address of the iSCSI initiator
• <iscsi-vmk> is the VMkernel adapter on the vSAN iSCSI network
• vmk0 is the management VMkernel adapter

1. Confirm the target daemon is listening. The output shows vitd in LISTEN state on both the iSCSI VMkernel address and the virtual IP address.

   esxcli network ip connection list | grep 3260

2. Confirm the ESXi firewall permits inbound TCP 3260. The output shows the rule enabled with Allowed IP set to All.

   esxcli network firewall ruleset rule list | grep 3260

3. Confirm basic reachability between the host and the initiator. Both directions succeed with no loss.

   vmkping -I <iscsi-vmk> <initiator-ip>

4. Establish a control on a different TCP port. A connection from the same initiator to a different TCP port on the same iSCSI VMkernel address completes a three-way handshake, while a connection to TCP 3260 times out.

5. Check the socket state during a failed attempt. The connection from the initiator appears in RCVD state, which shows the inbound SYN reaches the target service.

   esxcli network ip connection list | grep <initiator-ip>

6. Capture on the iSCSI VMkernel adapter during a failed attempt. The capture shows only inbound SYN packets and their retransmissions. No SYN-ACK leaves this adapter.

   tcpdump-uw -i <iscsi-vmk> -nn host <initiator-ip> and port 3260

7. Capture on the management VMkernel adapter during the same failed attempt. The capture shows the SYN-ACK leaving vmk0 sourced from the iSCSI VMkernel address on TCP port 3260. The packet retransmits several times with no acknowledgement.

   tcpdump-uw -i vmk0 -nn host <initiator-ip>

8. Confirm access control is not a factor. Verify whether iSCSI initiator groups are configured. When no initiator groups exist, the target accepts all initiators and access control is ruled out.

Environment

• VMware vSAN
• VMware vSphere ESXi
• vSAN iSCSI target service enabled
• iSCSI initiators located on a different IP subnet than the vSAN iSCSI VMkernel network

Cause

The ESXi host routing table contains no route to the initiator subnet through the iSCSI VMkernel gateway.

The vSAN iSCSI target service receives the inbound SYN and generates a SYN-ACK sourced from the iSCSI VMkernel address. Route selection for that reply uses the host routing table and matches the most specific prefix available. With no entry for the initiator subnet, the reply matches only the default route.

The default route next hop is reachable on the management VMkernel adapter. The SYN-ACK therefore egresses vmk0 carrying a source address that belongs to the iSCSI subnet. The reply does not reach the initiator, the initiator never sends the final ACK, and the half-open connection times out.

Configuring a gateway override on the iSCSI VMkernel adapter does not resolve this. The override does not add an entry for the initiator subnet to the host routing table, so the target service reply still matches the default route.

An initiator on the same subnet as the vSAN iSCSI network is unaffected. That reply matches a directly connected route and never consults the default route.

Resolution

Add a static route on every ESXi host in the vSAN cluster that directs traffic for the initiator subnet through the iSCSI VMkernel gateway.

Important: Apply the route to every host in the vSAN cluster, not only the host that currently owns the virtual IP address. The vSAN iSCSI target service redirects initial connections to the current I/O owner of each target. Both virtual IP ownership and target I/O ownership move between hosts. A route present on only one host results in discovery succeeding while login fails.

Substitute the values that apply to the environment:

• <iscsi-gateway-ip> is the gateway address on the vSAN iSCSI network
• <initiator-subnet/cidr> is the subnet the iSCSI initiator resides on

1. Connect to the ESXi host over SSH.

2. Add the static route.

   esxcli network ip route ipv4 add --gateway <iscsi-gateway-ip> --network <initiator-subnet/cidr>

3. Confirm the route is present. The output lists the initiator subnet with the iSCSI gateway as the next hop.

   esxcli network ip route ipv4 list

4. Repeat steps 1 through 3 on every remaining ESXi host in the vSAN cluster.

5. Repeat steps 1 through 3 for each additional initiator subnet that requires access to the vSAN iSCSI target service.

6. Retry discovery and login from the initiator. The target is discovered and the LUN mounts.

Routes added with esxcli network ip route ipv4 add persist across a host reboot.

The change is additive and narrowly scoped. It does not modify the default route or any connected route, and it affects only traffic destined for the initiator subnet. Management, vSAN, and vMotion traffic remain unchanged.

If the issue persists after following these steps, contact Broadcom Support for further assistance. Provide the following information when opening a support request with Broadcom for this issue:

• vSAN and ESXi log bundles from the affected hosts
• Output of esxcli network ip route ipv4 list from each host in the vSAN cluster
• Output of esxcli network ip interface ipv4 address list from the host that owns the virtual IP address
• Packet captures taken on both the iSCSI and management VMkernel adapters during a failed connection attempt
• The initiator address and subnet mask, and the addressing used on the vSAN iSCSI network