VMware vSphere Replication 5.1.x shows virtual machine status as: RPO violation for VMs - NFC code: 10 NFC Error: the parent of this virtual disk could not be opened. Can't open remote disk
search cancel

VMware vSphere Replication 5.1.x shows virtual machine status as: RPO violation for VMs - NFC code: 10 NFC Error: the parent of this virtual disk could not be opened. Can't open remote disk

book

Article ID: 307322

calendar_today

Updated On:

Products

VMware Live Recovery VMware vSphere ESXi

Issue/Introduction

This article provides information on diagnosing a specific SRM failure.

Symptoms:

  • VMware vSphere Replication tab displays the virtual machine status as:

    RPO violation for VMs - NFC code: 10 NFC Error: the parent of this virtual disk could not be opened. Can't open remote disk.

  • On the PR host site, you see these messages from hostd:

    2013-07-12T00:44:44.752Z [25B67B90 info 'Hbrsvc'] Replicator: Poweron for VM: (id=37) VM
    2013-07-12T00:44:44.764Z [25AA4B90 info 'ha-eventmgr' opID=366D5ADF-0000C21C-4e-b9] Event 7395 : Replication configuration changed for virtual machine VM on host host.domain in cluster host.domain in ha-datacenter (2 disks, 120 minutes RPO, VR Server is 10.20.2.12:44046).
    2013-07-12T00:44:44.784Z [25B67B90 verbose 'Hbrsvc'] Creating replication group for VM VM with spec: {dynamicType = <unset>, generation = 13, vmReplicationId = "GID-c0c0e22b-136f-####-####-########594", destination = "10.20.2.12", port = 44046, rpo = 120, quiesceGuestEnabled = true, paused = false, oppUpdatesEnabled = false, disk = [{dynamicType = <unset>, key = 2000, diskReplicationId = "RDID-289f0417-beb4-####-####-########025", }, {dynamicType = <unset>, key = 2001, diskReplicationId = "RDID-9468b35c-4f2f-####-####-########edf", }], }
    2013-07-12T00:44:44.786Z [25B67B90 verbose 'Hbrsvc'] Created replication group for VM: VM (GroupID=GID-c0c0e22b-136f-####-####-########594)
    2013-07-12T00:44:44.792Z [25B67B90 verbose 'Hbrsvc'] Replicator: VM 37 Disk (Key=2000,Id=RDID-289f0417-beb4-####-####-########025) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-289f0417-beb4-####-####-########025.psf
    2013-07-12T00:44:44.793Z [25AA4B90 verbose 'Hbrsvc' opID=366D5ADF-0000C21C-4e-b9] Replicator: VM 37 Disk (Key=2000,Id=RDID-289f0417-beb4-####-####-########025) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-289f0417-beb4-####-####-########025.psf
    2013-07-12T00:44:44.795Z [25B67B90 verbose 'Hbrsvc'] Replicator: VM 37 Disk (Key=2001,Id=RDID-9468b35c-4f2f-####-####-########edf) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-9468b35c-4f2f-####-####-########edf.psf
    2013-07-12T00:44:44.796Z [25AA4B90 verbose 'Hbrsvc' opID=366D5ADF-0000C21C-4e-b9] Replicator: VM 37 Disk (Key=2001,Id=RDID-9468b35c-4f2f-####-####-########edf) has psf: [SMD_###_03] VM/hbr-persistent-state-RDID-9468b35c-4f2f-####-####-########edf.psf
    2013-07-12T00:45:44.711Z [25BE9B90 info 'ha-eventmgr'] Event 7396 : No connection to VR Server for virtual machine VM on host host.domain in cluster host.domain in ha-datacenter: Unknown


  • In the HMS log on VR at DR, you see messages similar to these:

    2013-07-11 17:37:02.522 INFO hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [hms-vlsi-server-thread-76] (..response.filter.I18nActivationResponseFilter) operationID=471C
    1805-0001BE37 | The localized message is: Replication error encountered at VR Server for replication 'VM'. Details: 'Error for (datastoreUUID: "51795b26-a098####-####-########c4a"), (diskId:
    "RDID-289f0417-####-####-####-##########25"), (hostId: "host-18"), (pathname: "VM/hbrdisk.RDID-289f0417-beb4-####-####--#######025.2952.233849802062572.vmdk"), (flags: on-disk-open, retriable): Class: NFC Code: 10; NFC error: The parent of this virtual disk could not be opened; Set error flag: retriable; Can't open remote disk /vmfs/volumes/51795b26-########-####-##########4a/VM/hbrdisk.RDID-289f0417-####-####-####-##########25.2952.233849802062572.vmdk; Can't open remote disk /vmfs/volumes/51795b26-a098####-####-########c4a/VM/hbrdisk.RDID-289f0417-beb4-####-####-########025.2952.233849802062572.vmdk; Set error flag: on-disk-open; Tried operation 4 times, giving up.; Failed to open replica (/vmfs/volumes/51795b26-########-####-##########4a/VM/hbrdisk.RDID-289f0417-####-####-####-##########25.2952.233849802062572.vmdk); Failed to open activeDisk (GroupID=GID-c0c0e22b-136f-####-####-########594) (DiskID=RDID-289f0417-beb4-####-####-########025); Can't create replica state (GroupID=GID-c0c0e22b-136f-####-####-########594) (DiskID=RDID-289f0417-beb4-####-####-########025); Cannot activate group. Loading disks from database (GroupID=GID-
    c0c0e22b-136f-####-####-########594) '.
    2013-07-11 17:37:02.522 INFO hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [hms-vlsi-server-thread-76] (..response.filter.I18nActivationResponseFilter) operationID=471C
    1805-0001BE37 | The localized message is: Replication error encountered at VR Server for replication 'VM2'. Details: 'Error for (datastoreUUID: "51795b26-a098####-####-########c4a"), (diskId: "RDID-120cc845-d9d8-####-####-########d06"), (hostId: "host-16"), (pathname: "VM2/hbrdisk.RDID-########-####-####-####-##########06.2932.100336141226620.vmdk"), (flags: on-disk-open, retriable): Class: NFC Code: 10; NFC error: The parent of this virtual disk could not be opened; Set error flag: retriable; Can't open remote disk /vmfs/volumes/51795b26-a09####-####-########c4a/VM2/hbrdisk.RDID-120cc845-d9d8-4820-b195-ef273d524d06.2932.100336141226620.vmdk; Can't open remote disk /vmfs/volumes/51795b26-########-####-##########4a/VM2/hbrdisk.RDID-120cc845-####-####-####-##########06.2932.100336141226620.vmdk; Set error flag: on-disk-open; Tried operation 4 times, giving up.; Failed to open replica (/vmfs/volumes/51795b26-########-####-##########4a/VM2/hbrdisk.RDID-120cc845-####-####-####-##########06.2932.100336141226620.vmdk); Failed to open activeDisk (GroupID=GID-9da9053d-dc8d-####-####-########eb5) (DiskID=RDID-120cc845-d9d8-####-####-########d06); Can't create replica state (GroupID=GID-9da9053d-####-####-####-##########b5) (DiskID=RDID-120cc845-####-####-####-##########06); Cannot activate group. Loading disks from database (GroupID=GID-9da9053d-dc8d-####-####-########eb5) '.


  • You successfully issue the command nc -z to the IP address of the VR server on port 31031 and 44046, but in host on PR - vmkernel log, messages indicate a failure to establish a connection to R :44046.
  • Replicated base disks are found in a different datastore than expected.
  • SDRS is enabled on the cluster for the datastore group.
  • SDRS is configured for VR and ABR.
  • Error messages may indicate an issue with network communication from Primary Site to DR site from VR server. However, this is not a network issue.

Environment

VMware vSphere Replication 5.1.x

Cause

This issue occurs because SDRS is not supported for VMware Replication or Array-Based Replication in SRM.

For more information, see:

Resolution

To prevent this issue from occurring in the future, avoid using SDRS in an SRM environment in the future.

Warning: To resolve this issue, contact Broadcom Technical Support to work with an SRM-skilled engineer. For information on filing a support request, see Creating and managing Broadcom support cases.


Additional Information

Impact/Risks:
Warning: Attempting to resolve this issue without technical support may result in permanent data loss.