Normal operation on virtual machine fails with error: systemd-hostnamed.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-hostnamed: Input/output error

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• These symptoms are observed when using iSCSI connectivity with Dell Power flex storage.
• Network switch upgrades leading to network issues.
• Virtual machine normal operation may fail with I/O errors.
• Lost access to volumes.

Validation:

Force restarting a virtual machine ,ay encounter below errors:

Mar 25 21:59:41 TestVM systemd[3888243]: systemd-hostnamed.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-hostnamed: Input/output error

Mar 25 23:12:09 TestVM systemd[4147761]: Failed to resolve symlink /home/pamaccnt/.config/systemd/user.control, ignoring: Input/output error

Mar 25 23:12:09 TestVM systemd[4147761]: Failed to resolve symlink /home/pamaccnt/.config/systemd/user, ignoring: Input/output error

Mar 25 23:12:09 TestVM systemd[4147761]: Failed to resolve symlink /home/pamaccnt/.local/share/systemd/user, ignoring: Input/output error

Mar 25 23:12:09 TestVM systemd[4147777]: systemd-hostnamed.service:Failed to set up mount namespacing: /run/systemd/unit-root/home: Input/output error

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESX 8.x

Cause

Power Flex cluster can withstand only ONE node going down at a time. If more than one node goes down or becomes unavailable, then ESXI may lose the storage connectivity. Due to which VM may not function or normal operation may fail with Input/Errors error when attempting to use the attached hard disk.

Resolution

Work around:

Force Virtual machine reboot may fix this but however Host reboot is necessary to refresh the hard disk as the normal operation may fail.

 

As per storage vendor, Fault Sets feature can be used to avoid these issues when more than one SDS becomes unavailable:

Engage the storage vendor to further investigate the issue.

Additional Information

1.Fault Sets are logical entities that contain a group of SDSs within a protection domain. A Fault Set is defined as a set of servers likely to fail together, for example, an entire rack full of servers.

2.PowerFlex requires a minimum of three Fault Sets per protection domain, with at least two nodes in each Fault Set.

3.Each fault set must have at least two nodes to achieve high availability and tolerance.

4.Fault Set is a logical entity that contains a group of SDSs within a Protection Domain, that have a higher chance of going offline together, such as a group of SDSs that are all powered in the same rack. By grouping them into a Fault Set, you are configuring PowerFlex to mirror the data for all devices in this Fault Set. The mirroring should take place on SDSs that are outside of this Fault Set.

 

Below are the possible causes for Lost access to volume when using iSCSI connectivity with power flex:

1. Some arrays are using virtual IP replacing the real IPs of the array controller, so try to replace it with the real IP.

2. Even though the LUN might be accessible, an MTU mismatch can affect the datastore creation on it.

3. SCSI initiator IQN was entered with the upper-case letters,

4. SDC is not installed

5. SDC is not attached to the cluster

6. Volume was not mapped to SDC

7 Volume was not mapped to the SCSI initiator

8. SVM IP was not entered into the dynamic discovery IPs under the iSCSI adapter properties

9. SVM network adapter is not on the same network as the VMkernel

 

Respective vendors can be engaged to check below:

1.Verify that the storage array is listed in the Broadcom compatibility guide and that the initiator is registered on the array. Consult your storage vendor for instructions on this procedure.

2.Verify that the array is configured correctly for use with your ESXi hosts. Partner with your hardware vendor to ensure that the array is properly configured. 

3.Verify that the physical hardware and physical network hardware are functioning correctly. You may have to contact your hardware vendor for information on verifying that your hardware functions correctly.