Quiesced snapshot based backups of Aria Automation or Orchestrator appliance fails with 'Freeze synchronization failed' message during execution of /opt/scripts/freezer-server.py script
book
Article ID: 343821
calendar_today
Updated On:
Products
VMware Aria Suite
Issue/Introduction
Symptoms:
The snapshot fails with an error similar to:
The guest OS has reported an error during quiescing. The error code was: 2 The error message was: Custom quiesce script failed.
When checking the journal logs on the Aria Automation or Orchestrator appliance using the journalctl command you see log entries similar to:
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: Traceback (most recent call last):
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: File "/etc/vmware-tools/backupScripts.d/90-freeze-data", line 166, in <module>
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: sys.exit(main(sys.argv))
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: File "/etc/vmware-tools/backupScripts.d/90-freeze-data", line 152, in main
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: [arg0, op] = argv
Jun 27 04:10:43 <vRA-FQDN> vmtoolsd[698]: ValueError: too many values to unpack (expected 2)
The issue occurs due to how the freezer-control.py script handles the arguments being passed to it by VMTools when the snapshot has been requested by specific backup solutions.
Resolution
This issue is resolved in VMware Aria Automation or Orchestrator 8.13 and above.
Workaround:
Prerequisites
You have backups of the VMware Aria Automation or Orchestrator 8.x appliance(s)
You must back up all VMware Aria Automation or Orchestrator appliances, at the same time - simultaneously for all nodes.
If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
When you back up the VMware Aria Automation or Orchestrator appliance, disable in-memory snapshots and enable quiescing (quiescing is a requirement only for version 8.9 and newer).
Note: Since quiescing is impacted, run /opt/scripts/deploy.sh --shutdown, followed by powering off the cluster and create cold powered off snapshots as an alternative.
You have access to root user and password.
Procedure
SSH / PuTTy into one node in the cluster.
Run the following command:
vracli cluster exec -- sh -c "base64 -d <<< '/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AEdANtdADgYSu7FLY3pHPHC/W5O/novQtmAzFq4I9ABTNjc3eS0G9c7871dPCp6hMBQ+09cywgIysB1ITZOL6QqIQI4WMysTYScZUtkBDckajdTXO8M9hZP6+IC+nLUpejTBj4UsJAJvGwDEF1hL2GhNZd6U9Ah8PwNVbjZqjurpnNiXOKgZtzK0gIvlE5pDGZzUYtKD1boSV4Mw1Dx1b9XJHBix+R6SZkbr52O9BP1x4L6VyXkq0wdnaosa5RCD8aF5AogUT37OOEi4LMjmvGyO/Q94+ylxxySJM7KUKeAAAAAxCF3nKW7GogAAfcBngIAADWWuaixxGf7AgAAAAAEWVo=' | xz -d | sh -"
Additional Information
In some cases, snapshots can fail due to incorrect VMware tools configuration. The configuration file is /etc/vmware-tools/tools.conf and its content should be:
[vmbackup] execScripts=true # Avoid deadlock (our scripts thaw `/` too late) enableXmlManifest=false # Exclude / and /data as we freeze them and tools cannot excludedFileSystems=/,/data