Corfu data file corruption seen in corfu.9000.log or corfu.9040.log: "Checksum mismatch detected while trying to read file" or "Can't parse metadata"
search cancel

Corfu data file corruption seen in corfu.9000.log or corfu.9040.log: "Checksum mismatch detected while trying to read file" or "Can't parse metadata"

book

Article ID: 303324

calendar_today

Updated On:

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

  • DATASTORE or CORFU_NONCONFIG service is in degraded/down state

  • Following errors may be seen in NSX Manager logs:

    • /var/log/corfu/corfu.9000.log shows error

      <Time-stamp>| ERROR | WrapperSimpleAppMain | o.c.infrastructure.CorfuServer | Failed starting server
      <Time-stamp> | ERROR | WrapperSimpleAppMain | o.c.infrastructure.CorfuServer | Failed starting server org.corfudb.runtime.exceptions.DataCorruptionException: Can't parse metadata. Segment File: /config/corfu/log/297779.log. File size: 3872130. File position: 3871666
      ...
      ...
      Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

      or

      <Time-stamp> | ERROR | WrapperSimpleAppMain | o.c.infrastructure.CorfuServer | CorfuServer: Server exiting due to unrecoverable error: org.corfudb.runtime.exceptions.DataCorruptionException: Checksum mismatch detected while trying to read file
    • /var/log/corfu/corfu-compactor-audit.log may show message "Tried to get layout from <node with corruption IP>:9000 but failed by timeout"

    • /var/log/syslog reports the below error

      <Time-stamp> nsx-mgr01 NSX 3489 - - getClusterStatus: Error while fetching layout from 10.##.##.##:9000. Exception: 
      <Time-stamp> nsx-mgr01 NSX 2342 - - Connect Async 10.##.##.##:9000
    • /var/log/corfu-nonconfig/corfu.9040.log shows the following error

      org.corfudb.runt ime.exceptions.DataCorruptionException: Can't parse metadata. Segment File: /nonconfig/corfu/corfu/log/60.log. File size: 528425. File position: 515050

Environment

VMware NSX

Cause

When the Corfu server starts up, it loads binary data files and verifies checksums. If a data file is corrupted, Corfu cannot recover on its own. Having a cluster of Corfu nodes (3 nodes) protects against this scenario. If Corfu data files are corrupted on a node, then the node needs to be removed and replaced with a new node.

The issue can also occur in Greenfield VCF deployments if there are underlying storage issues on the host(s) where the NSX Managers reside.

Resolution

If only one NSX Manager node shows signs of Corfu corruption:

Remove and replace corrupted Manager node:

  1. Record the IP and FQDN of the Manager node with corrupted data

  2. Confirm if the NSX Manager VM to be redeployed is the orchestrator node:

    • get service install-upgrade

  3. If the node to be replaced is the orchestrator, change the orchestrator to a manager appliance that is not being replaced (NOTE: This command needs to be run from one of the NSX Manager nodes that you are not going to replace/detach.)

    • set repository-ip

  4. Record configuration settings for node to be replaced

  5. To find the node UUID of the NSX Manager to be replaced

    • get nodes

  6. Detach the node from the cluster via admin CLI of a node not being replaced

    • detach node <UUID>

  7. Check cluster status to ensure the node has been removed from all cluster services

    • get cluster status

  8. Power off detached node and delete from disk via vSphere UI

  9. Deploy new NSX Manager Appliance via NSX UI System > Appliances > Add NSX Appliance

  10. If the NSX Manager node needs to be manually deployed, use the following process:

  11. Wait for repo_sync to complete after cluster stabilizes in the NSX UI (this process can take some time)

Alternatively:

  • If the failed NSX Manager was auto-deployed through the NSX UI, the corrupt NSX Manager can be deleted in the NSX Manager UI under System > Appliances and the Delete option under Actions for the corrupt node. Then a new NSX Manager node can deployed to return the cluster to three nodes. 

If two NSX Manager nodes in a three node cluster is showing indications of Corfu corruption:

Reduce the cluster to a single node of only the working NSX manager and replace the defective nodes.

  1. Power off the degraded virtual appliances in the vSphere Client.

  2. From the CLI of the healthy surviving node, ensure the node is the repository master:

    • set repository-ip

  3. Deactivate the cluster to make this the only cluster node
    • deactivate cluster

  4. Verify that the node is the only node in the cluster:

    • get cluster status

  5. Remove (delete from disk) the previously deactivated and powered off nodes from vCenter.

  6. Log into the NSX Manager UI, navigate to System > Appliances, and deploy two clean replacement NSX Manager nodes matching the original network configuration.

  7. Allow the newly deployed appliances to join the cluster, synchronize the Corfu database, and restore the 3-node cluster quorum.

If all three NSX Manager nodes have Corfu data files that are corrupted:

  • The only recourse is to restore from valid NSX Backup. Reference documentation at Restore a Backup