Best practices for backup and restore related issues on SFD
Article ID: 323693
Updated On:
Products
VMware
Issue/Introduction
Symptoms:
Enabling backup and restore via CLI on SFD and verifying debug logs when issue arises
Enabling backup and restore via CLI on SFD and verifying debug logs when issue arises
Resolution
Configuring and debugging Backup/restore related issues in SFD
sfd> backup location list
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| # | IP Address/FQDN | User | Backup Directory | Protocol | Created By | Created At(UTC) |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| 1 | 10.79.xx.xx | ftpuser | /FTP/files | FTP | [email protected] | 2021-01-18 10:30:12 |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
sfd> backup location add --host 10.79.xx.xx --user <username> --password <password> --backupdir /FTP/files --protocol ftp
sfd> backup job schedule --name backup_1 --content all --description first_backup
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| # | IP Address/FQDN | User | Backup Directory | Protocol | Created By | Created At(UTC) |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| 1 | 10.79.xx.xx | ftpuser | /FTP/files | FTP | [email protected] | 2021-01-18 10:30:12 |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
Please enter number of backup location that you want to store backup data: 1
Backup job scheduling status: BACKUP_JOB_OPERATION_SUCCESS
sfd> backup job list
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
| # | Job Name | Status | Location | Recurrence | Content | Created By | Created At(UTC) |
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
| 1 | backup_1 | Expired | ftp://10.79.xx.xx/FTP/files | Never | Configurations | [email protected] | 2021-01-18 11:08:45 |
| | | | | | Events and Logs | | |
| | | | | | Monitoring Data | | |
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
Below is an example of backup with logs
Initiating Backup
2021/01/19 02:44:30.579 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:429) - Start new backup activity
2021/01/19 02:44:30.579 [pool-6-thread-5] [DEBUG] com.vmware.nfc.platform.activitymanager.ActivityManager.createActivity(ActivityManager.java:146) - Received request to create new activity for name backup2 message Start taking new backup: 'backup2' for backup job: 'backup2_cem_19-04-2021_01-44-30.557.tar'. type NFC_ACTIVITY_TYPE_BACKUP subtype NFC_ACTIVITY_SUB_TYPE_CREATE_BACKUP status NFC_ACTIVITY_STATUS_IN_PROGRESS startTime 1618800270579 refOid oid: "208:53044847265649914"
with 4 step
2021/01/19 02:44:30.586 [pool-6-thread-5] [DEBUG] com.vmware.nfc.platform.activitymanager.ActivityPublisher.updateAndPublishActivity(ActivityPublisher.java:51) - Receiving request to update oid {
oid: "190:53044847266370811"
}
config {
type: NFC_ACTIVITY_TYPE_BACKUP
sub_type: NFC_ACTIVITY_SUB_TYPE_CREATE_BACKUP
start_time: 1618800270579
status: NFC_ACTIVITY_STATUS_IN_PROGRESS
name: "backup2"
message: "Start taking new backup: \'backup2\' for backup job: \'backup2_cem_18-01-2021_02-44-30.557.tar\'."
service_name: "system-controller-service"
current_step: -1
total_step: 4
}
refs {
back_refs {
reference_oid {
oid: "208:53044847265649914"
}
}
}
activity with message: Starting backup... on BACKUP_ACTIVITY channel
2021/01/19 02:44:30.587 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:435) - Set current backup instance status to IN_PROGRESS
2021/01/19 02:44:30.622 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:440) - Start taking new backup
2021/01/19 02:44:30.622 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.runBackup(NfcBackupOrchestrator.java:316) - contentTypes [CONFIG_DATA, EVENTS_AND_LOGS_DATA, MONITORING_DATA]
Completion of backup
2021/01/19 02:44:42.396 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.eventmanager.AbstractEventPublisher.publish(AbstractEventPublisher.java:111) - PUBLISH: Publish BACKUP_UPLOAD_SUCCESS_EVENT event on channel : BACKUP_UPLOAD_STATUS
2021/01/19 02:44:42.433 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.handler.BackupJobHandler.setBackupJobExpiry(BackupJobHandler.java:325) - Update the backup job state as COMPLETED
Restore Command
sfd> backup restore initiate --filepath ftp://10.79.xx.xx/FTP/files/first-backup_ce_12-01-2021_06-18-25.571.tar --username <usernmae> --password <Password>
Retrieving backup file info
Retrieved Backup File Attributes
+-------------------------------+---------------------------------------------+
| Name | Value |
+-------------------------------+---------------------------------------------+
| File Name | first-backup_ce_12-01-2021_06-18-25.571.tar |
| File Size | 34 MB |
| File Modified Timestamp (UTC) | 2021-01-12 06:16:42 |
+-------------------------------+---------------------------------------------+
Initiate restore request
Initiate restore status: RESTORE_INITIATED
Once restore is complete we should be able to see below messages
2021-01-19 03:17:46,421 [INFO ][services:sfd.common.services:verify_service_stop] - Pods have stopped!
2021-01-19 03:17:46,631 [INFO ][main:root:stop_app] - shut down maintenance app
2021-01-19 03:17:48,426 [INFO ][main:root:__start_services] - Stopped nginx in maintenance mode and maintenace app
2021-01-19 03:17:48,426 [INFO ][services:sfd.common.services:start_service] - Starting service /opt/vmware/nfc/scripts/deployment/nginx-gw.yaml
2021-01-19 03:17:48,587 [INFO ][services:sfd.common.services:verify_service_start] - Verifying pods startup for service label[application=nginx-gw]
2021-01-19 03:17:48,697 [INFO ][services:sfd.common.services:verify_service_start] - Pods are not ready. Validating after 15 secs
2021-01-19 03:18:03,821 [INFO ][services:sfd.common.services:verify_service_start] - Pods are ready!
2021-01-19 03:18:03,822 [INFO ][main:root:init_restore] - ---------------restore success!-----------------
2021-01-19 03:18:03,822 [INFO ][main:root:init_restore] - Nginx started
During backup or restore if there are any failures we will need to collect the logs in debug and can verify in the below location
sfd> backup location list
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| # | IP Address/FQDN | User | Backup Directory | Protocol | Created By | Created At(UTC) |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| 1 | 10.79.xx.xx | ftpuser | /FTP/files | FTP | [email protected] | 2021-01-18 10:30:12 |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
sfd> backup location add --host 10.79.xx.xx --user <username> --password <password> --backupdir /FTP/files --protocol ftp
sfd> backup job schedule --name backup_1 --content all --description first_backup
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| # | IP Address/FQDN | User | Backup Directory | Protocol | Created By | Created At(UTC) |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
| 1 | 10.79.xx.xx | ftpuser | /FTP/files | FTP | [email protected] | 2021-01-18 10:30:12 |
+---+-----------------+---------+------------------+----------+-----------------+---------------------+
Please enter number of backup location that you want to store backup data: 1
Backup job scheduling status: BACKUP_JOB_OPERATION_SUCCESS
sfd> backup job list
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
| # | Job Name | Status | Location | Recurrence | Content | Created By | Created At(UTC) |
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
| 1 | backup_1 | Expired | ftp://10.79.xx.xx/FTP/files | Never | Configurations | [email protected] | 2021-01-18 11:08:45 |
| | | | | | Events and Logs | | |
| | | | | | Monitoring Data | | |
+---+----------+---------+-------------------------------+------------+-----------------+-----------------+---------------------+
Below is an example of backup with logs
Initiating Backup
2021/01/19 02:44:30.579 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:429) - Start new backup activity
2021/01/19 02:44:30.579 [pool-6-thread-5] [DEBUG] com.vmware.nfc.platform.activitymanager.ActivityManager.createActivity(ActivityManager.java:146) - Received request to create new activity for name backup2 message Start taking new backup: 'backup2' for backup job: 'backup2_cem_19-04-2021_01-44-30.557.tar'. type NFC_ACTIVITY_TYPE_BACKUP subtype NFC_ACTIVITY_SUB_TYPE_CREATE_BACKUP status NFC_ACTIVITY_STATUS_IN_PROGRESS startTime 1618800270579 refOid oid: "208:53044847265649914"
with 4 step
2021/01/19 02:44:30.586 [pool-6-thread-5] [DEBUG] com.vmware.nfc.platform.activitymanager.ActivityPublisher.updateAndPublishActivity(ActivityPublisher.java:51) - Receiving request to update oid {
oid: "190:53044847266370811"
}
config {
type: NFC_ACTIVITY_TYPE_BACKUP
sub_type: NFC_ACTIVITY_SUB_TYPE_CREATE_BACKUP
start_time: 1618800270579
status: NFC_ACTIVITY_STATUS_IN_PROGRESS
name: "backup2"
message: "Start taking new backup: \'backup2\' for backup job: \'backup2_cem_18-01-2021_02-44-30.557.tar\'."
service_name: "system-controller-service"
current_step: -1
total_step: 4
}
refs {
back_refs {
reference_oid {
oid: "208:53044847265649914"
}
}
}
activity with message: Starting backup... on BACKUP_ACTIVITY channel
2021/01/19 02:44:30.587 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:435) - Set current backup instance status to IN_PROGRESS
2021/01/19 02:44:30.622 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.initiateBackup(NfcBackupOrchestrator.java:440) - Start taking new backup
2021/01/19 02:44:30.622 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.NfcBackupOrchestrator.runBackup(NfcBackupOrchestrator.java:316) - contentTypes [CONFIG_DATA, EVENTS_AND_LOGS_DATA, MONITORING_DATA]
Completion of backup
2021/01/19 02:44:42.396 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.eventmanager.AbstractEventPublisher.publish(AbstractEventPublisher.java:111) - PUBLISH: Publish BACKUP_UPLOAD_SUCCESS_EVENT event on channel : BACKUP_UPLOAD_STATUS
2021/01/19 02:44:42.433 [pool-6-thread-5] [INFO ] com.vmware.nfc.platform.systemcontroller.backupandrestore.handler.BackupJobHandler.setBackupJobExpiry(BackupJobHandler.java:325) - Update the backup job state as COMPLETED
Restore Command
sfd> backup restore initiate --filepath ftp://10.79.xx.xx/FTP/files/first-backup_ce_12-01-2021_06-18-25.571.tar --username <usernmae> --password <Password>
Retrieving backup file info
Retrieved Backup File Attributes
+-------------------------------+---------------------------------------------+
| Name | Value |
+-------------------------------+---------------------------------------------+
| File Name | first-backup_ce_12-01-2021_06-18-25.571.tar |
| File Size | 34 MB |
| File Modified Timestamp (UTC) | 2021-01-12 06:16:42 |
+-------------------------------+---------------------------------------------+
Initiate restore request
Initiate restore status: RESTORE_INITIATED
Once restore is complete we should be able to see below messages
2021-01-19 03:17:46,421 [INFO ][services:sfd.common.services:verify_service_stop] - Pods have stopped!
2021-01-19 03:17:46,631 [INFO ][main:root:stop_app] - shut down maintenance app
2021-01-19 03:17:48,426 [INFO ][main:root:__start_services] - Stopped nginx in maintenance mode and maintenace app
2021-01-19 03:17:48,426 [INFO ][services:sfd.common.services:start_service] - Starting service /opt/vmware/nfc/scripts/deployment/nginx-gw.yaml
2021-01-19 03:17:48,587 [INFO ][services:sfd.common.services:verify_service_start] - Verifying pods startup for service label[application=nginx-gw]
2021-01-19 03:17:48,697 [INFO ][services:sfd.common.services:verify_service_start] - Pods are not ready. Validating after 15 secs
2021-01-19 03:18:03,821 [INFO ][services:sfd.common.services:verify_service_start] - Pods are ready!
2021-01-19 03:18:03,822 [INFO ][main:root:init_restore] - ---------------restore success!-----------------
2021-01-19 03:18:03,822 [INFO ][main:root:init_restore] - Nginx started
During backup or restore if there are any failures we will need to collect the logs in debug and can verify in the below location
- Backup logs can be found in the system-controller-service and NFC-host-service
- Restore logs can be found in system-controller-service, NFC-host-service and restore.log file
Feedback
Yes
No
Powered by
