Upgrading host to ESXi 6.5.0 build-7388607 fails with PSOD

Products

• VMware vSAN

Issue/Introduction

Symptoms:
Upgrading a host to ESXi 6.5.0 build-7388607 fails with the purple diagnostic screen.

2018-02-03T22:43:50.421Z cpu8:66965)@BlueScreen: Failed at bora/modules/vmkernel/vsan/dom/domOwner.c:2974 -- NOT REACHED
2018-02-03T22:43:50.422Z cpu8:66965)Code start: 0x41800f200000 VMK uptime: 0:00:02:13.357
2018-02-03T22:43:50.422Z cpu8:66965)0x43916ca9b990:[0x41800f2ed531]PanicvPanicInt@vmkernel#nover+0x545 stack: 0x41800f2ed531
2018-02-03T22:43:50.422Z cpu8:66965)0x43916ca9ba30:[0x41800f2ed8c7]Panic_vPanic@vmkernel#nover+0x23 stack: 0x0
2018-02-03T22:43:50.423Z cpu8:66965)0x43916ca9ba50:[0x41800f301501]vmk_PanicWithModuleID@vmkernel#nover+0x41 stack: 0x43916ca9bab0
2018-02-03T22:43:50.423Z cpu8:66965)0x43916ca9bab0:[0x418010667e29][email protected]#0.0.0.1+0x1fd stack: 0x49e580
2018-02-03T22:43:50.423Z cpu8:66965)0x43916ca9bb00:[0x41801068908a][email protected]#0.0.0.1+0x416 stack: 0xd
2018-02-03T22:43:50.424Z cpu8:66965)0x43916ca9bdf0:[0x418010689ab1][email protected]#0.0.0.1+0x715 stack: 0x200000000
2018-02-03T22:43:50.424Z cpu8:66965)0x43916ca9bed0:[0x41801025def2][email protected]#0.0.0.1+0x51a stack: 0x430c0e96f4e8
2018-02-03T22:43:50.424Z cpu8:66965)0x43916ca9bf90:[0x41800f308bbf]vmkWorldFunc@vmkernel#nover+0x4f stack: 0x41800f308bba
2018-02-03T22:43:50.425Z cpu8:66965)0x43916ca9bfe0:[0x41800f4c93c5]CpuSched_StartWorld@vmkernel#nover+0x99 stack: 0x0
2018-02-03T22:43:50.427Z cpu8:66965)base fs=0x0 gs=0x418042000000 Kgs=0x0
2018-02-03T22:43:50.427Z cpu8:66965)2 other PCPUs are in panic.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 6.0.x
VMware vSAN 6.6.x
VMware vSAN 6.2.x
VMware vSAN 6.1.x
VMware vSAN 6.5.x

Cause

This issue occurs if the unicast agent configuration of a node is added to its own unicast agent list.

Resolution

This issue is resolved in vSAN 6.6.1 U2.

Workaround:
To workaround this issue, perform the steps below:

List the unicast agents with the command:

esxcli vsan cluster unicastagent list

Note: This should give the list of the neighbor nodes as well as the local node, in case you find local node details, please remove the node from the unicastagent list.
 
esxcli vsan cluster unicastagent remove -a xx.xx.xx.xx -u host uuid.

Note: Once the workaround is applied, make sure you never add the unicast info of the local node to its own unicast agent list.