Red Hat Enterprise Linux 6 or 7 virtual machine running on ESXi 6 fails with VMCISock errors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Red Hat Enterprise Linux 6 or 7 virtual machine running on ESXi 6 fails.
• Virtual machine fails with VMCISock errors.
• You see the stack trace:
  
  RHEL 6:
  
  <stacktrace>
  [<ffffffff8105e646>] ? enqueue_task+0x66/0x80
  [<ffffffff8105e683>] activate_task+0x23/0x30
  [<ffffffff810670c0>] try_to_wake_up+0x1f0/0x3e0
  [<ffffffff810672c2>] default_wake_function+0x12/0x20
  [<ffffffff81059939>] __wake_up_common+0x59/0x90
  [<ffffffff8105e168>] __wake_up+0x48/0x70
  [<ffffffffa0148e40>] ? vmci_event_unsubscribe+0x114/0x154 [vmci]
  [<ffffffffa0148e64>] vmci_event_unsubscribe+0x138/0x154 [vmci]
  [<ffffffffa013f992>] VMCI_WaitOnEventInterruptible+0x7e/0xc0 [vmci]
  [<ffffffff810672b0>] ? default_wake_function+0x0/0x20
  [<ffffffff8107fe48>] ? irq_exit+0x48/0x90
  [<ffffffffa0148da8>] vmci_event_unsubscribe+0x7c/0x154 [vmci]
  [<ffffffffa017db60>] VMCISock_KernelDeregister+0x54c/0x704 [vsock]
  [<ffffffff8145b303>] __sk_free+0x23/0x170
  [<ffffffff8145b4d5>] sk_free+0x25/0x30
  [<ffffffffa0180411>] VSockVmciAllowDgram+0x1e11/0x2e44 [vsock]
  [<ffffffffa0145759>] ? VMCIHashTable_GetEntry+0xa5/0xb8 [vmci]
  [<ffffffffa0142130>] VMCIDatagram_InvokeGuestHandler+0xf0/0x1cc [vmci]
  [<ffffffffa0141ce7>] VMCI_ReadDatagramsFromPort+0xd3/0x158 [vmci]
  [<ffffffff810804e5>] tasklet_action+0xe5/0x120
  [<ffffffff8107ffd1>] __do_softirq+0xc1/0x1e0
  [<ffffffff810ed980>] ? handle_IRQ_event+0x60/0x170
  [<ffffffff8100c38c>] call_softirq+0x1c/0x30
  [<ffffffff8100fbd5>] do_softirq+0x65/0xa0
  [<ffffffff8107fe85>] irq_exit+0x85/0x90
  [<ffffffff815424e5>] do_IRQ+0x75/0xf0
  [<ffffffff8100ba53>] ret_from_intr+0x0/0x11
  <EOI>
  [<ffffffff8104301b>] ? native_safe_halt+0xb/0x10
  [<ffffffff810167ed>] default_idle+0x4d/0xb0
  [<ffffffff81009fe6>] cpu_idle+0xb6/0x110
  [<ffffffff81531932>] start_secondary+0x2c0/0x316
  </Stacktrace>
  
  RHEL 7:
  
  <Stacktrace>
  [201414.815949] <IRQ> [<ffffffff815e19ba>] dump_stack+0x19/0x1b
  [201414.815961] [<ffffffff815dbfe9>] __schedule_bug+0x4d/0x5b
  [201414.815966] [<ffffffff815e718b>] __schedule+0x78b/0x790
  [201414.815970] [<ffffffff815e71b9>] schedule+0x29/0x70
  [201414.815975] [<ffffffff815e50b9>] schedule_timeout+0x209/0x2d0
  [201414.815981] [<ffffffff8108ec58>] ? __wake_up_common+0x58/0x90
  [201414.815985] [<ffffffff81090a24>] ? __wake_up+0x44/0x50
  [201414.815990] [<ffffffff815e76e6>] wait_for_completion+0x116/0x170
  [201414.815995] [<ffffffff81097700>] ? wake_up_state+0x20/0x20
  [201414.816001] [<ffffffff810ff6a0>] ? __call_rcu+0x2c0/0x2c0
  [201414.816008] [<ffffffff810826ab>] wait_rcu_gp+0x5b/0x80
  [201414.816012] [<ffffffff81082630>] ? ftrace_raw_output_rcu_utilization+0x50/0x50
  [201414.816016] [<ffffffff8108ec58>] ? __wake_up_common+0x58/0x90
  [201414.816020] [<ffffffff810fef5b>] synchronize_sched+0x3b/0x50
  [201414.816027] [<ffffffffa031a9d0>] vmci_event_unsubscribe+0x70/0xb0 [vmw_vmci]
  [201414.816032] [<ffffffffa032f611>] vmci_transport_destruct+0x21/0xe0
  [vmw_vsock_vmci_transport]
  [201414.816037] [<ffffffffa03a0867>] vsock_sk_destruct+0x17/0x60 [vsock]
  [201414.816044] [<ffffffff814bb73f>] __sk_free+0x1f/0x170
  [201414.816048] [<ffffffff814bb8b1>] sk_free+0x21/0x30
  [201414.816052] [<ffffffffa03302bb>] vmci_transport_recv_stream_cb+0x1fb/0x2e0
  [vmw_vsock_vmci_transport]
  [201414.816058] [<ffffffff810bd9a4>] ? tick_program_event+0x24/0x30
  [201414.816063] [<ffffffffa0319f1c>] vmci_datagram_invoke_guest_handler+0xbc/0xf0
  [vmw_vmci]
  [201414.816069] [<ffffffffa031ae6f>] vmci_dispatch_dgs+0xcf/0x230 [vmw_vmci]
  [201414.816075] [<ffffffff81066eae>] tasklet_action+0x6e/0x110
  </Stacktrace>
  
  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs when the Virtual Machine Communication Interface runs an incorrect call and returns a NULL pointer.

Resolution

This is a known issue affecting ESXi 6.0. To resolve this issue:

 

Red Hat Enterprise Linux 6:

 

To resolve this issue for RHEL 6 operating systems, install the VMware Tools that is included in ESXi 6.0 Update 1 and later available at VMware Downloads. For more information, see VMware vCenter Server 6.0 Update 1 Release Notes.

 

Red Hat Enterprise Linux 7:

 

To resolve this issue for RHEL 7 operating systems, update the Red Hat kernel to:

• For RHEL7.3, this issue is resolved in kernel-3.10.0-514.el7
• For RHEL7.2z, this issue is resolved in kernel-3.10.0-327.10.1.el7
• For RHEL7.1z, this issue is resolved in kernel-3.10.0-229.28.1.el7

Additional Information

Using ESXi Shell in ESXi 5.x and 6.x
ESXI 6 で実行している Red Hat Enterprise Linux 6 または 7 の仮想マシンが VMCISock エラーで失敗する
在 ESXI 6 上运行的 Red Hat Enterprise Linux 6 或 7 虚拟机发生故障并出现 VMCISock 错误