Quiesced snapshot based backups of Aria Automation or Orchestrator appliance fails with 'Freeze synchronization failed' message during execution of /opt/scripts/freezer-server.py script

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• The snapshot fails with an error similar to:

  The guest OS has reported an error during quiescing. The error code was: 2 The error message was: Custom quiesce script failed.

• When checking the journal logs on the Aria Automation or Orchestrator appliance using the journalctl command you see log entries similar to:

  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: Traceback (most recent call last):
  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]:   File "/etc/vmware-tools/backupScripts.d/90-freeze-data", line 166, in <module>
  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]:     sys.exit(main(sys.argv))
  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]:   File "/etc/vmware-tools/backupScripts.d/90-freeze-data", line 152, in main
  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]:     [arg0, op] = argv
  Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: ValueError: too many values to unpack (expected 2)

Environment

VMware Aria Automation Orchestrator 8.x
VMware Aria Automation 8.x
VMware vRealize Automation 8.x
VMware vRealize Orchestrator 8.x

Cause

The issue occurs due to how the freezer-control.py script handles the arguments being passed to it by VMTools when the snapshot has been requested by specific backup solutions.

Resolution

This issue is resolved in VMware Aria Automation or Orchestrator 8.13 and above.

Workaround:

Prerequisites

• You have backups of the VMware Aria Automation or Orchestrator 8.x appliance(s)
  • You must back up all VMware Aria Automation or Orchestrator appliances, at the same time - simultaneously for all nodes.
  • If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
  • When you back up the VMware Aria Automation or Orchestrator appliance, disable in-memory snapshots and enable quiescing (quiescing is a requirement only for version 8.9 and newer).

Note: Since quiescing is impacted, run /opt/scripts/deploy.sh --shutdown, followed by powering off the cluster and create cold powered off snapshots as an alternative.

• You have access to root user and password.

Procedure

1. SSH / PuTTy into one node in the cluster.
2. Run the following command:

vracli cluster exec -- sh -c "base64 -d <<< '/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AEdANtdADgYSu7FLY3pHPHC/W5O/novQtmAzFq4I9ABTNjc3eS0G9c7871dPCp6hMBQ+09cywgIysB1ITZOL6QqIQI4WMysTYScZUtkBDckajdTXO8M9hZP6+IC+nLUpejTBj4UsJAJvGwDEF1hL2GhNZd6U9Ah8PwNVbjZqjurpnNiXOKgZtzK0gIvlE5pDGZzUYtKD1boSV4Mw1Dx1b9XJHBix+R6SZkbr52O9BP1x4L6VyXkq0wdnaosa5RCD8aF5AogUT37OOEi4LMjmvGyO/Q94+ylxxySJM7KUKeAAAAAxCF3nKW7GogAAfcBngIAADWWuaixxGf7AgAAAAAEWVo=' | xz -d | sh -" 

Additional Information

In some cases, snapshots can fail due to incorrect VMware tools configuration. The configuration file is /etc/vmware-tools/tools.conf and its content should be:

[vmbackup]
execScripts=true
# Avoid deadlock (our scripts thaw `/` too late)
enableXmlManifest=false
# Exclude / and /data as we freeze them and tools cannot
excludedFileSystems=/,/data

[guestinfo]
exclude-nics=lo,docker*,veth*,flannel*,cni*,dummy*,kube-ipvs*

[powerops]
poweron-script=/opt/vmware/share/vami/vami_poweron_vm_script
resume-script=/opt/vmware/share/vami/vami_resume_vm_script