VMware vSphere Replication 5.1.x shows virtual machine status as: RPO violation for VMs - NFC code: 10 NFC Error: the parent of this virtual disk could not be opened. Can't open remote disk

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

This article provides information on diagnosing a specific SRM failure.

Symptoms:

VMware vSphere Replication tab displays the virtual machine status as:

RPO violation for VMs - NFC code: 10 NFC Error: the parent of this virtual disk could not be opened. Can't open remote disk.
On the PR host site, you see these messages from hostd:

2013-07-12T00:44:44.752Z [25B67B90 info 'Hbrsvc'] Replicator: Poweron for VM: (id=37) VM
2013-07-12T00:44:44.764Z [25AA4B90 info 'ha-eventmgr' opID=366D5ADF-0000C21C-4e-b9] Event 7395 : Replication configuration changed for virtual machine VM on host host.domain in cluster host.domain in ha-datacenter (2 disks, 120 minutes RPO, VR Server is 10.20.2.12:44046).
2013-07-12T00:44:44.784Z [25B67B90 verbose 'Hbrsvc'] Creating replication group for VM VM with spec: {dynamicType = <unset>, generation = 13, vmReplicationId = "GID-c0c0e22b-136f-####-####-########594", destination = "10.20.2.12", port = 44046, rpo = 120, quiesceGuestEnabled = true, paused = false, oppUpdatesEnabled = false, disk = [{dynamicType = <unset>, key = 2000, diskReplicationId = "RDID-289f0417-beb4-####-####-########025", }, {dynamicType = <unset>, key = 2001, diskReplicationId = "RDID-9468b35c-4f2f-####-####-########edf", }], }
2013-07-12T00:44:44.786Z [25B67B90 verbose 'Hbrsvc'] Created replication group for VM: VM (GroupID=GID-c0c0e22b-136f-####-####-########594)
2013-07-12T00:44:44.792Z [25B67B90 verbose 'Hbrsvc'] Replicator: VM 37 Disk (Key=2000,Id=RDID-289f0417-beb4-####-####-########025) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-289f0417-beb4-####-####-########025.psf
2013-07-12T00:44:44.793Z [25AA4B90 verbose 'Hbrsvc' opID=366D5ADF-0000C21C-4e-b9] Replicator: VM 37 Disk (Key=2000,Id=RDID-289f0417-beb4-####-####-########025) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-289f0417-beb4-####-####-########025.psf
2013-07-12T00:44:44.795Z [25B67B90 verbose 'Hbrsvc'] Replicator: VM 37 Disk (Key=2001,Id=RDID-9468b35c-4f2f-####-####-########edf) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-9468b35c-4f2f-####-####-########edf.psf
2013-07-12T00:44:44.796Z [25AA4B90 verbose 'Hbrsvc' opID=366D5ADF-0000C21C-4e-b9] Replicator: VM 37 Disk (Key=2001,Id=RDID-9468b35c-4f2f-####-####-########edf) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-9468b35c-4f2f-####-####-########edf.psf
2013-07-12T00:45:44.711Z [25BE9B90 info 'ha-eventmgr'] Event 7396 : No connection to VR Server for virtual machine VM on host host.domain in cluster host.domain in ha-datacenter: Unknown
In the HMS log on VR at DR, you see messages similar to these:

2013-07-11 17:37:02.522 INFO hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [hms-vlsi-server-thread-76] (..response.filter.I18nActivationResponseFilter) operationID=471C
1805-0001BE37 | The localized message is: Replication error encountered at VR Server for replication 'VM'. Details: 'Error for (datastoreUUID: "51795b26-a098####-####-########c4a"), (diskId:
"RDID-289f0417-####-####-####-##########25"), (hostId: "host-18"), (pathname: "VM/hbrdisk.RDID-289f0417-beb4-####-####--#######025.2952.233849802062572.vmdk"), (flags: on-disk-open, retriable): Class: NFC Code: 10; NFC error: The parent of this virtual disk could not be opened; Set error flag: retriable; Can't open remote disk /vmfs/volumes/51795b26-########-####-##########4a/VM/hbrdisk.RDID-289f0417-####-####-####-##########25.2952.233849802062572.vmdk; Can't open remote disk /vmfs/volumes/51795b26-a098####-####-########c4a/VM/hbrdisk.RDID-289f0417-beb4-####-####-########025.2952.233849802062572.vmdk; Set error flag: on-disk-open; Tried operation 4 times, giving up.; Failed to open replica (/vmfs/volumes/51795b26-########-####-##########4a/VM/hbrdisk.RDID-289f0417-####-####-####-##########25.2952.233849802062572.vmdk); Failed to open activeDisk (GroupID=GID-c0c0e22b-136f-####-####-########594) (DiskID=RDID-289f0417-beb4-####-####-########025); Can't create replica state (GroupID=GID-c0c0e22b-136f-####-####-########594) (DiskID=RDID-289f0417-beb4-####-####-########025); Cannot activate group. Loading disks from database (GroupID=GID-
c0c0e22b-136f-####-####-########594) '.
2013-07-11 17:37:02.522 INFO hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [hms-vlsi-server-thread-76] (..response.filter.I18nActivationResponseFilter) operationID=471C
1805-0001BE37 | The localized message is: Replication error encountered at VR Server for replication 'VM2'. Details: 'Error for (datastoreUUID: "51795b26-a098####-####-########c4a"), (diskId: "RDID-120cc845-d9d8-####-####-########d06"), (hostId: "host-16"), (pathname: "VM2/hbrdisk.RDID-########-####-####-####-##########06.2932.100336141226620.vmdk"), (flags: on-disk-open, retriable): Class: NFC Code: 10; NFC error: The parent of this virtual disk could not be opened; Set error flag: retriable; Can't open remote disk /vmfs/volumes/51795b26-a09####-####-########c4a/VM2/hbrdisk.RDID-120cc845-d9d8-4820-b195-ef273d524d06.2932.100336141226620.vmdk; Can't open remote disk /vmfs/volumes/51795b26-########-####-##########4a/VM2/hbrdisk.RDID-120cc845-####-####-####-##########06.2932.100336141226620.vmdk; Set error flag: on-disk-open; Tried operation 4 times, giving up.; Failed to open replica (/vmfs/volumes/51795b26-########-####-##########4a/VM2/hbrdisk.RDID-120cc845-####-####-####-##########06.2932.100336141226620.vmdk); Failed to open activeDisk (GroupID=GID-9da9053d-dc8d-####-####-########eb5) (DiskID=RDID-120cc845-d9d8-####-####-########d06); Can't create replica state (GroupID=GID-9da9053d-####-####-####-##########b5) (DiskID=RDID-120cc845-####-####-####-##########06); Cannot activate group. Loading disks from database (GroupID=GID-9da9053d-dc8d-####-####-########eb5) '.
You successfully issue the command nc -z to the IP address of the VR server on port 31031 and 44046, but in host on PR - vmkernel log, messages indicate a failure to establish a connection to R :44046.
Replicated base disks are found in a different datastore than expected.
SDRS is enabled on the cluster for the datastore group.
SDRS is configured for VR and ABR.
Error messages may indicate an issue with network communication from Primary Site to DR site from VR server. However, this is not a network issue.

Environment

VMware vSphere Replication 5.1.x

Cause

This issue occurs because SDRS is not supported for VMware Replication or Array-Based Replication in SRM.

For more information, see:

Resolution

To prevent this issue from occurring in the future, avoid using SDRS in an SRM environment in the future.

Warning: To resolve this issue, contact Broadcom Technical Support to work with an SRM-skilled engineer. For information on filing a support request, see Creating and managing Broadcom support cases.

Additional Information

Impact/Risks:
Warning: Attempting to resolve this issue without technical support may result in permanent data loss.