VM crashes during the restart of guest OS with error: "Systemd[1] : sysinit. target: Unable to break cycle starting with sysinit.target/start"

Products

• VMware vSAN

Issue/Introduction

• Restarting the guest operating system results in a VM crash with below error:

"Systemd[1] : sysinit. target: Unable to break cycle starting with sysinit.target/start".

• In vCenter UI, Cluster > Monitor > vSAN > Skyline Health might report the MTU Check errors for vSAN.

• In vCenter UI, Host > Monitor > Triggered Alarms shows error as High pNic error rate detected.

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

Issue with the network connectivity between the hosts using vSAN network.

• vmkping between the hosts using vSAN network reports packet loss.

[root@Hostname:~] vmkping -I vmk2  ###.###.###.## -s 1472
PING ###.###.###.## (###.###.###.##): 1472 data bytes
1480 bytes from ###.###.###.##: icmp_seq=1 ttl=64 time=0.143 ms
1480 bytes from ###.###.###.##: icmp_seq=2 ttl=64 time=0.098 ms

--- ###.###.###.## ping statistics ---
3 packets transmitted, 2 packets received, 33.3333% packet loss
round-trip min/avg/max = 0.098/0.120/0.143 ms

 

• High CRC errors are symptoms of the network issues. It could be a defective cable, SFP, or Network Interface Card (NIC) etc.

High PNIC errors can also be validated in /var/run/log/hostd.log

 

2025-02-19T00:48:20.265Z Wa(164) Hostd[2100138]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic0
2025-02-19T00:48:20.265Z Wa(164) Hostd[2100079]: --> droppedRx: 22597
2025-02-19T00:48:20.266Z Wa(164) Hostd[2100079]: --> errorsRx: 22597
2025-02-19T00:48:20.266Z Wa(164) Hostd[2100079]: --> RxCRCErrors: 22597

CRC errors with Rx receive and dropped messages can also be checked in VMNIC stats

esxcli network nic stats get –n <vmnic interface>

   Total receive errors: 1494198013
   Receive length errors: 12975260
   Receive over errors: 0
   Receive CRC errors: 1137854000

Resolution

Engage the network team/vendor to investigate the vSAN network connectivity issue between the hosts.

Additional Information

CRC:

The CRC stands for "Cyclic Redundancy Check". The FCS (Frame Check Sequence) field contains a 4-byte CRC value used for error checking. When a source host assembles a packet, it performs a CRC calculation on all fields in the packet except the Preamble, SFD (Start Frame Delimiter), and FCS using a predetermined algorithm. The source host stores the value in the FCS field and transmits it as part of the packet. When the packet is received by the destination host, it performs a CRC test again by using the same algorithm. If the CRC value calculated at the destination host does not match the value in the FCS field, the destination host discards the packet, considering this as a CRC Error.

Troubleshooting vSAN Networking

Alarm about high pNIC error rate being detected

View vSAN Network Alarms