VMware vSphere Data Protection 5.1 manual integrity check fails with the error: An integrity check could not be performed at this time, please try again at a later time
search cancel

VMware vSphere Data Protection 5.1 manual integrity check fails with the error: An integrity check could not be performed at this time, please try again at a later time

book

Article ID: 307858

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Running a manual integrity check from the Configuration tab in the vSphere Data Protection (VDP) plugin fails.
  • Running a manual integrity check from the VDP plugin user interface times out.
  • You see the error:

    An integrity check could not be performed at this time, please try again at a later time.

  • The vdr-server.log file (located at /usr/local/avamar/var/vdr/server_logs/ ) contains entries similar to:
13:02:41,287 ERROR [http-8543-exec-5]-service.AdapterUtils: Server failed to create checkpoint for server "xxxxxxxxx"

com.avamar.mc.sdk.client.RuntimeFaultMsgException: DPNSTORAGE_CHECKPOINT_CREATE_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx timeout: Creating checkpoint
at com.avamar.mc.sdk.client.McTaskWorker.doInBackground(McTaskWorker.java:72)
at com.avamar.mc.sdk.client.McTaskWorker.access$000(McTaskWorker.java:23)
at com.avamar.mc.sdk.client.McTaskWorker$1.call(McTaskWorker.java:56)
at com.avamar.mc.sdk.client.McTaskWorker$1.call(McTaskWorker.java:52)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


Environment

VMware vSphere Data Protection 5.1.x

Cause

This issue occurs when a manual integrity check is run from the VDP plugin user interface and does not complete within one minute.

Resolution

To work around this issue, run a manual integrity check from the VDP plug-in user interface at a time of low network traffic or disk I/O.

Caution: Running an integrity check is highly CPU intensive and can cause performance issues in VDP.
If manual integrity checks from the VDP plugin user interface continue to fail, log in to the VDP appliance with SSH to troubleshoot the issue:
  1. Check the current VDP status by running the command:

    status.dpn

    You see output similar to:

    Last checkpoint: cp.20130624204437 finished Mon Jun 24 14:46:25 YYYY after 01m 48s (OK)
    Last GC: finished Mon Jun 24 08:06:09 YYYY after 00m 09s >> recovered 4.83 MB (OK)
    Last hfscheck: finished Mon Jun 24 14:32:13 YYYY after 12m 31s >> checked 65 of 65 stripes (OK)


    Note: Check that the Last checkpoint and the Last hfscheck have finished.

  2. Stop scheduled services to ensure backups to not start by running the command:

    avmaint sched stop --ava

    The output reports no errors:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <sched-stop-results timestamp="1372107243">
    <response<br>code="MSG_ERR_NONE"
    nodeid="0.0"/>
    </sched-stop-results>

  3. Check that the checkpoint is the most recent submission and shows as completed by running the command:

    avmaint checkpoint --ava

    The output reports no errors:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <checkpoint<br>tag="cp.CCYYMMDD######"
    isvalid="false"/>

  4. Check again that hfscheck process is not running using the command:

    status.dpn

  5. Run a full integrity check again using the command:

    avmaint hfscheck --ava --full

    You see output similar to:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <hfscheck<br>checkpoint="cp.CCYYMMDD#######"
    status="waitcgsan"
    type="full"
    checks="full"
    elapsed-time="79"
    start-time="1372107460"
    end-time="0"
    check-start-time="0"
    check-end-time="0"
    generation-time="1372107539"
    percent-complete="0.00">
    <hfscheckerrors/> <<< NO ERRORS HERE<br></hfscheck>

    Note: While the integrity check runs, the command line session to the VDP appliance may stop responding. To check the hfscheck process, open a second SSH session to the VDP appliance, and run the status.dpn command again. You see output listing the hfscheck process:

    Hfscheck in progress: started Mon Jun 01 14:57:40 YYYY (####)

    Run the status.dpn command until you see output indicating that the hfscheck process is finished:

    Last hfscheck: finished Mon Jun 24 15:03:29 YYYY after 05m 49s >> checked 65 of 65 stripes (OK)

  6. Check that the latest checkpoint output contains the valid and hfs flags by running the command:

    cplist

    You see output similar to:

    cp.20130601204437 Mon Jun 01 14:44:37 YYYY valid --- --- nodes 1/1 stripes 66
    cp.20130601205502 Mon Jun 01 14:55:02 YYYY valid hfs --- nodes 1/1 stripes 66

  7. Check that the hfscheck status displays as completed for the latest checkpoint by running the command:

    avmaint hfscheckstatus --ava

    You see output similar to:

    checkpoint="cp.20130601205502"
    status="completed"

  8. Start the scheduled services by running the command:

    avmaint sched start --ava


Additional Information

For further information, see Running a manual integrity check in vSphere Data Protection (VDP) (2067701).
Running a manual integrity check in VMware vSphere Data Protection (VDP)
VMware vSphere Data Protection 5.1 の手動での整合性チェックが失敗し、「この時点で整合性チェックを実行することはできませんでした。後でもう一度実行してください」というエラーが発生します。
VMware vSphere Data Protection 5.1 手动完整性检查失败,并显示错误“目前无法执行完整性检查,请稍后重试”

Impact/Risks:
Running an integrity check is highly CPU intensive and can cause performance issues in VDP. Run an integrity check at a time of low network traffic or disk I/O.