Rubrik backup fails when Rubrik CDP is enabled on the VM.

Products

• VMware vCenter Server

Issue/Introduction

• Rubrik backup fails for VMs with the Rubrik CDP (Continuous Data Protection) feature enabled and works fine for VMs with Rubrik CDP disabled.
• The backup operations of snapshot creation and deletion work as expected for VMs with the Rubrik CDP disabled, but take an unusually large amount of time for VMs with the Rubrik CDP enabled.
• vCenter is healthy.
  • You may have upgraded vCenter recently.
• In the var/run/log/iofilterd-rubiofilter.log of the ESXi host where the impacted VM(s) reside, you see error messages similar to the ones below:

iofilterd-rubiofilter[4520160]: InitThriftTransport:269: instance 96B716FDF8D03E8822C961E36F66787B thrift error : failed to connect to host 10.###.##.000:1514 - Connection timed out
iofilterd-rubiofilter[4520160]: InitThriftTransport:269: instance 96B716FDF8D03E8822C961E36F66787B thrift error : failed to connect to host 10.###.##.001:1514 - Connection timed out
iofilterd-rubiofilter[4520160]: InitThriftTransport:269: instance 96B716FDF8D03E8822C961E36F66787B thrift error : failed to connect to host 10.###.##.002:1514 - Connection timed out
iofilterd-rubiofilter[4520160]: InitThriftTransport:269: instance 96B716FDF8D03E8822C961E36F66787B thrift error : failed to connect to host 10.###.##.003:1514 - Connection timed out
iofilterd-rubiofilter[4520160]: InitThriftTransport:269: instance 96B716FDF8D03E8822C961E36F66787B thrift error : failed to connect to host 10.###.##.004:1514 - Connection timed out
iofilterd-rubiofilter[4520160]: CtrlcThriftDiskOpen:464: instance 96B716FDF8D03E8822C961E36F66787B disk open failed, thrift: (result = Ok, error = Ok), cdp status = Error

Cause

Port 1514 is closed on the Network for at least some IPs used for the communication between vSphere and Rubrik. In the example shown in the introduction, the IPs that have the port closed appear in the log error messages (10.###.##.000, 10.###.##.001)...

Resolution

There are 2 ways to address this issue:

 

1. Correct the settings related to Network in question, opening port 1514 for vCenter / Rubrik traffic for the IPs reporting the problems.

2. Check the Rubrik Storage policy in use to see if the IPs configured in the policy are the expected ones. If not, edit the policy to replace them with the correct IP addresses.