Manually Backing Up and Restoring Redis for Pivotal Cloud Foundry
search cancel

Manually Backing Up and Restoring Redis for Pivotal Cloud Foundry

book

Article ID: 295148

calendar_today

Updated On:

Products

Redis for VMware Tanzu

Issue/Introduction

This article describes how to manually back up and restore Redis for PCF.


Environment


Resolution

Redis can be manually backed up and restored in PCF. This process involves some steps that are mentioned below one by one:
 

Step 1. Find the Redis Deployment Name

To identify your Redis for the PCF deployment name, do the following:

  1. In the Operations Manager installation dashboard, click the Redis for PCF tile.
  2. The deployment name is the part of the URL starting with p-redis and ending with a GUID.

 

Step 2. Identify and SSH onto the Redis Instance you want to Back Up

Select one of the following procedures depending on the type of service plan you have:

On-Demand Service Instance

  1. Follow these steps to SSH onto your Operations Manager installation.

  2. Identify the service instance GUID by running cf service SERVICE_INSTANCE_NAME --guid.
    Here, SERVICE_INSTANCE_NAME     is the name chosen when running the cf create service command. 

  3. Using the GUID you identified above, SSH onto your service instance by running bosh2 -d service-instance_GUID ssh.

Note: The bosh2 commands are only relevant to PCF 1.12 and earlier. As of PCF 2.0, you can start the command with bosh. 

Dedicated-VM-service-instance

  1. Follow these steps to SSH onto your Operations Manager installation.

  2. Retrieve the deployment name of the Redis for PCF tile.

In order to identify the dedicated node that has the instance you want to back up, use the deployment name you retrieved above in this command:

bosh2 -d REDIS-DEPLOYMENT-NAME vms

The example output lists several dedicated node instances:

bosh -d cf-redis vms
Using environment 'https://100.155.99.99:99999' as client 'ops_manager'

Task 12550. Done

Deployment 'cf-redis'

Instance Process State AZ IPs VM CID VM Type cf-redis-broker/example-d4444-474c-9a44-e4144cbc4ec4 running europe-west1-b 10.0.0.0 vm-7835sample-44b2-4544-544d-8d4fb21d0bb3 medium dedicated-node/1f444771-69bb-416b-8ded-a4409b3f0e7b running europe-west1-b 10.0.0.0 vm-4d55sampleb-44ac-4efd-44b5-a0d1444bccb0 medium dedicated-node/644b829f-ba43-4cf3-9f44-644caade44ec running europe-west1-b 10.0.0.0 vm-31f1ea1b-444e-4aa5-44fe-db44446044ae medium dedicated-node/c44df62d-38bb-44a5-9d44-8c44414f6e53 running europe-west1-b 10.0.0.0 vm-3d44c1b8-44ae-4e98-4c44-8f6f6ec444ce medium

4 vms

Succeeded

If you don’t know which dedicated-node instance to use, follow these steps:

  1. Do one of the following:
     
    • If you have an app bound to the dedicated instance, run cf env APP-NAME
       
    • If you do not have an app bound to the dedicated instance, run the following command to create a service key:
      cf create-service-key SERVICE-INSTANCE-NAME mykey && cf service-key SERVICE-INSTANCE-NAME mykey
  2. In the output of the previous command, there should be an IP address shown that corresponds to an IP address output from the bosh2 vms command. Match this IP to the matching dedicated-node/GUID in the output shown in Step 3.

  3. Use that GUID from the previous step and the Redis deployment name to SSH onto the VM:
    bosh2 -d REDIS-DEPLOYMENT-NAME ssh dedicated-node/GUID


Step 3. Make a manual backup 

You can create a snapshot which creates an RDB file locally or you can create a snapshot and send it to the remote backup destination.

  • To create a snapshot only, run: sudo /var/vcap/jobs/redis-backups/bin/backup --snapshot. This creates a new RDB file in /var/vcap/store/redis-backup

  • To create a snapshot and send it to the pre-configured remote backup destination, run sudo /var/vcap/jobs/service-backup/bin/manual-backup.
    You must have enabled and pre-configured this destination in the "Backups" tab of the Redis for PCF settings. 

    Note: A manual backup of a large dump.rdb file can take so long that your SSH connection times out. Therefore, when configuring the backup destination, ensure that you have given yourself enough time in the "Backup timeout" field to complete the backup.

 

Step 4. Restore a Redis for PCF Instance from a Backup

You can also restore your backup file to another instance of the Redis for PCF tile. It can be the same or a different plan.

Prerequisites for Restore

  • The instance you are restoring to must have the same resource configuration as the instance you backed up.

  • Ensure that the persistent disk is large enough to handle the temporary files used during the restore process. It should be 3.5x the amount of RAM in the VM.

Preparation

  1. Create a new instance of the plan that you want to restore to.

  2. Identify the bosh instance which you just created

  3. Bosh scp the backup file into the /tmp directory
    For example: bosh -d cf-redis scp ~/Downloads/2017-12-18.rdb dedicated-node/236203b2-1afa-4a57-9a75-218f068f2d6d:/tmp/2017-12-18.rdb

  4. SSH onto the VM which the instance of your plan is located on.

  5. On this VM, create a directory under /var/vcap/store, for example my-backups.

  6. Copy the RDB file from the /tmp directory to the directory you created above, or in any directory below it. Substitute this path, not including var/vcap/store in the next step.

  7. To verify that the RDB file has not been corrupted run:

    md5sum PATH-TO-RDB-FILE

    For example: md5sum my-backups/2017-12-18.rdb

  8. Compare the value returned in the above step against the contents of the .md5 file that has a similar name to the backup file. The values should be the same.

    The .md5 file is located in the same location as the original backup file.

  9. Switch to root user using sudo su.
     

Restore to an On-Demand or Dedicated-VM Plan

Run the command:

sudo /var/vcap/jobs/redis-backups/bin/restore --sourceRDB PATH-TO-RDB-FILE

 

Restore to a Shared-VM Plan

  1. Retrieve the GUID for the instance you want to restore to by running:

    cf service SERVICE-INSTANCE-NAME --guid

  2. Use the GUID you found above in this command:

    sudo /var/vcap/jobs/redis-backups/bin/restore --sourceRDB PATH-TO-RDB-FILE --sharedVmGuid GUID


Step 5. Debugging restore

If the restore fails partway through, you may need to manually run some commands to get the instance back into a healthy state.

It logs its process along the way and provides helpful messages in case of failure. The logs are at /var/vcap/sys/log/redis-backups/redis-backups.log.

The sections below list the restore process steps and related messages that appear on your console. If a step fails, resolve the reason that caused it to fail, and then execute the failed step and every subsequent step manually.

You can retrieve the INSTANCE-PASSWORD through the binding to your service instance:

Note: The above debugging steps require running as a superuser.

Restore Process Steps for On-Demand and Dedicated-VM Instances

  1. StopAll
    Run monit stop all

  2. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary

  3. DeleteExistingPersistenceFiles
    Clean up existing Redis data files:

    • rm -f /var/vcap/store/redis/appendonly.aof
    • rm -f /var/vcap/store/redis/dump.rdb

  4. CopyBackupFileWithCorrectPermissions
    Restore your Redis backup file to /var/vcap/store/redis/dump.rdb and correct the owner and permissions with chown vcap:vcap /var/vcap/store/redis/dump.rdb && chmod 660 /var/vcap/store/redis/dump.rdb

  5. SetAppendOnly
    Edit the template Redis config file with vim $(find /var/vcap/data/jobs/ -name redis.conf)` and make the following line changes:

    • appendonly yes -> appendonly no

  6. StartAll
    Run monit start all

  7. WaitForStart
    Wait for monit services to enter the running state, you can watch this with watch monit summary

  8. RewriteAOF
    Run /var/vcap/packages/redis/bin/redis-cli -a INSTANCE-PASSWORD BGREWRITEAOF

  9. RewriteAOF
    Run watch "/var/vcap/packages/redis/bin/redis-cli -a INSTANCE-PASSWORD INFO | grep aof_rewrite_in_progress" until aof_rewrite_in_progress is 0

  10. StopAll
    Run monit stop all

  11. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary

  12. ChownToUserAndGroup
    Set correct owner on appendonly.aof by running chown vcap:vcap /var/vcap/store/redis/appendonly.aof

  13. SetAppendOnly
    Edit the template Redis config file with vim $(find /var/vcap/data/jobs/ -name redis.conf) and make the following line changes:

    • appendonly no -> appendonly yes

  14. StartAll
    Run monit start all

Restore Process Steps for Shared-VM Instances

  1. StopAll
    Run monit stop all

  2. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary

  3. SetConfigCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and comment out the line:

    • rename-command CONFIG "configalias" -> #rename-command CONFIG "configalias"

  4. SetRewriteCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and comment out the line:

    • rename-command BGREWRITEAOF "" -> #rename-command BGREWRITEAOF ""

  5. DeleteExistingPersistenceFiles
    Clean up existing Redis data files if they exist:

    • rm -f /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/db/appendonly.aof
    • rm -f /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/db/dump.rdb

  6. CopyBackupFileWithCorrectPermissions
    Restore your Redis backup file to /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/db/dump.rdb and correct the owner and permissions with chown vcap:vcap /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/db/dump.rdb && chmod 660 /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/db/dump.rdb

  7. SetAppendOnly
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and make the following line changes:

    • appendonly yes -> appendonly no

  8. StartAll
    Run monit start all

  9. WaitForStart
    Wait for monit services to enter the running state, you can watch this with watch monit summary

  10. RewriteAOF
    Run /var/vcap/packages/redis/bin/redis-cli -a INSTANCE-PASSWORD BGREWRITEAOF

  11. RewriteAOF
    Run watch "/var/vcap/packages/redis/bin/redis-cli -a INSTANCE-PASSWORD INFO | grep aof_rewrite_in_progress"until aof_rewrite_in_progress is 0

  12. StopAll
    Run monit stop all

  13. WaitForStop
    Wait for monit services to enter the not monitored state, you can watch this with watch monit summary

  14. ChownToUserAndGroup
    Set correct owner on appendonly.aof by running chown vcap:vcap /var/vcap/store/redis/appendonly.aof

  15. SetAppendOnly
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and make the following line changes:

    • appendonly no -> appendonly yes

  16. SetConfigCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and uncomment the line: #rename-command CONFIG "configalias" -> rename-command CONFIG "configalias"

  17. SetRewriteCommand
    Edit the template Redis config file with vim /var/vcap/store/cf-redis-broker/redis-data/INSTANCE-GUID/redis.conf and uncomment the line: #rename-command BGREWRITEAOF "" -> rename-command BGREWRITEAOF ""

  18. StartAll
    Run monit start all


Step 6. Restore an RDB File to Any Redis Server

You can choose to restore the RDB file to any Redis instance outside of PCF.

For information on how Redis handles RDB files, see Redis Persistence.


Step 7. Recover Redis Instances While Recovering PCF

In the event of a recovery of PCF, it is possible to recover bound Redis instances to healthy states that are in sync with PCF. There are a few requirements for fully recovering a previous instance state, depending on your plan.

Shared-VM Recovery Requirements

  • A backup Redis RDB dump file

  • A backed up /var/vcap/store/cf-redis-broker/redis-data directory from the service broker node. You do not need to back up *.aof or *.rdb files from subdirectories if you have backups configured.

Dedicated-VM Recovery Requirements

  • A backup Redis RDB dump file

  • A backed up /var/vcap/store/redis/statefile.json from the service broker node

On-Demand Recovery Requirements

  • A backup Redis RDB dump file

Recovery Procedure

Note: The procedures below assume that a recovery of service information and service keys assigned to instances are restored with a restore of PCF.

Recover a Shared-VM Plan

  1. bosh2 ssh into the service broker node of your Redis deployment.
  2. Run monit stop all && pkill redis-server.
  3. Wait for monit services to enter the not monitored state. You can watch this with watch monit summary.
  4. Confirm there are no running instances of redis-server by running ps aux | grep redis-server.
  5. Copy the backed up redis-data directory into /var/vcap/store/cf-redis-broker.
  6. Follow the instructions in Restore a Redis for PCF Instance from a Backup above for your plan, to restore your backed up Redis data.
  7. Your Redis instance is now recovered.

Recover a Dedicated-VM Plan

  1. bosh2 ssh into the service broker node of your Redis deployment.

  2. Run monit stop all

  3. Wait for monit services to enter the not monitored state, you can watch this with watch monit summary

  4. Copy the backed up /var/vcap/store/cf-redis-broker/statefile.json and ensure ownership and permissions are correct with
    chown vcap:vcap /var/vcap/store/cf-redis-broker/statefile.json && chmod 660 /var/vcap/store/cf-redis-broker/statefile.json

  5. Follow the instructions in Restore a Redis for PCF Instance from a Backup for your plan, to restore your backed up Redis data.

  6. Your Redis instance is now recovered.

Recover an On-Demand Plan

  1. Identify your target service-instance.
  2. Restore your backup file.

If your previous on-demand service-instance deployment does not exist:

  1. Create a new service-instances and restore your backup file.
  2. Clean up any old service-instances and bind the new service-instance to your app.


Additional Information

 


Powered by Wolken Software