How to setup pgBackrest to S3-Compatible Object Store
Article ID: 293288
Updated On:
Products
Issue/Introduction
With VMware Postgres, it is possible to store the backup files locally, in a remote server or in an S3 compatible object store.
With VMware Postgres for K8s, the built in backup solution leverages pgBackrest to perform the backups and those backups require an S3 compatible object store.
This article will describe how to setup these 2 products with Amazon S3 with the Tanzu Support AWS Account. Please follow up with your Regional Lead on the details of how to login to the Support AWS Account.
Environment
Resolution
Prerequisite: In the AWS S3 console, ensure that a bucket is created. This bucket will store the archive and backup folders which are used by pgBackrest to store archive logs and the physical backup files respectively.
The example below uses the bucket names "tanzu-postgres" and "tanzu-k8s" under "us-west-1" region with no folders to start off with. It is possible to make the bucket private by checking the "Block all Public Access" check mark under permissions.
If AWS CLI is installed in your local machine, you can test connectivity or your endpoint URL by uploading a file to the bucket using the aws s3 cp command.
Below are the settings you must use for each product:
VMware Postgres
In /etc/pgbackrest/pgbackrest.conf:
[global] repo1-retention-full=2 start-fast=y repo-path=/ repo1-type=s3 repo1-s3-key=<aws access key> repo1-s3-key-secret=<aws secret key> repo1-s3-bucket=tanzu-postgres repo1-s3-endpoint=s3.us-west-1.amazonaws.com repo1-s3-verify-tls=n repo1-s3-region=us-west-1 repo1-s3-uri-style=host
Create the stanza and run:
pgbackrest --stanza=<stanza name> --log-level-console=info check
If the above command is successful, verify the tanzu-postgres bucket to see if the files are there.
VMware Postgres for K8s (As of version 1.6.1)
In backuplocation.yaml:
apiVersion: sql.tanzu.vmware.com/v1
kind: PostgresBackupLocation
metadata:
name: backuplocation-sample
spec:
retentionPolicy:
fullRetention:
type: count
number: 9999999
diffRetention:
number: 9999999
storage:
s3:
bucket: "tanzu-k8s"
bucketPath: "/"
region: "us-west-1"
endpoint: "s3.us-west-1.amazonaws.com"
forcePathStyle: false
enableSSL: false
secret:
name: backuplocation-creds-sample
---
apiVersion: v1
kind: Secret
metadata:
name: backuplocation-creds-sample
type: generic
stringData:
accessKeyId: "<aws access key>"
secretAccessKey: "<aws secret key>"
1.) kubectl apply -f backuplocation.yaml -n default
2.) In your postgres.yaml or whichever file you used to create the postgres-sample pods, enable backups by uncommenting this section:
backupLocation:
name: backuplocation-sample
3.) kubectl apply -f postgres.yaml -n default
4.) Exec into the primary node. In this case, it is postgres-sample-1:
kubectl exec -ti pod/postgres-sample-1 -- bash
5.) Run the pgbackrest check command and make sure the command completes ok and the files are there in the tanzu-k8s bucket on S3.
postgres@postgres-sample-1:/$ pgbackrest --stanza=default-postgres-sample --log-level-console=info check 2022-04-13 17:30:17.751 P00 INFO: check command begin 2.38: --config=/etc/pgbackrest/pgbackrest.conf --exec-id=1211780-d0c134d8 --log-level-console=info --log-level-file=off --pg1-path=/pgsql/data --pg1-socket-path=/pgsql/custom/tmp --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/ --repo1-s3-bucket=#### --repo1-s3-endpoint=s3.us-west-1.#####.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-west-1 --repo1-s3-uri-style=host --no-repo1-storage-verify-tls --repo1-type=s3 --stanza=default-postgres-sample 2022-04-13 17:30:18.357 P00 INFO: check repo1 configuration (primary) 2022-04-13 17:30:18.799 P00 INFO: check repo1 archive for WAL (primary) 2022-04-13 17:30:19.604 P00 INFO: WAL segment 000000120000000000000026 successfully archived to '/archive/default-postgres-sample/14-1/0000001200000000/000000120000000000000026-#######.gz' on repo1 2022-04-13 17:30:19.604 P00 INFO: check command end: completed successfully (1854ms)
Once you get this message, the setup is complete and you can follow the instructions in our documentation to perform the actual backup and restore https://docs.vmware.com/en/VMware-Tanzu/index.html
Troubleshooting:
If you encounter the following error in either of the products, it means you are using the wrong endpoint/region/bucket in your config. In this case the endpoint and bucket name were incorrect however a connection is still able to be made. The problem is this URL causes a redirect which causes this specific command to fail. This check command must work in order for the backup to work too:
postgres@postgres-sample-1:/$ pgbackrest --stanza=default-postgres-sample --log-level-console=info check 2022-04-12 23:44:40.003 P00 INFO: check command begin 2.38: --config=/etc/pgbackrest/pgbackrest.conf --exec-id=2083-aa26f4be --log-level-console=info --log-level-file=off --pg1-path=/pgsql/data --pg1-socket-path=/pgsql/custom/tmp --repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc --repo1-path=/####### --repo1-s3-bucket=s3-us-west-1 --repo1-s3-endpoint=######.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-west-1 --repo1-s3-uri-style=host --no-repo1-storage-verify-tls --repo1-type=s3 --stanza=default-postgres-sample
2022-04-12 23:44:40.611 P00 INFO: check repo1 configuration (primary)
2022-04-12 23:44:40.911 P00 INFO: check repo1 archive for WAL (primary)
ERROR: [082]: WAL segment 0000000C000000000000001B was not archived before the 60000ms timeout
HINT: check the archive_command to ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log for errors.
HINT: run the 'start' command if the stanza was previously stopped.
2022-04-12 23:45:41.409 P00 INFO: check command end: aborted with exception [082]
A different combination may give you this error message as well. This occurs when the endpoint URL us invalid:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning NoSuitablePod 90s (x16 over 4m15s) postgres-backup-controller exit status 2 Warning Failed 34s postgres-backup-controller WARN: environment contains invalid option 'config-version'ERROR: [049]: unable to load info file '/####/my-bucket-path/archive/default-postgres-sample/archive.info' or '/####/my-bucket-path/archive/default-postgres-sample/archive.info.copy': HostConnectError: unable to get address for '####.#####.com': [-2] Name or service not known HostConnectError: unable to get address for '####.#####.com': [-2] Name or service not known HINT: archive.info cannot be opened but is required to push/get WAL segments. HINT: is archive_command configured correctly in postgresql.conf? HINT: has a stanza-create been performed? HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.
Feedback
