Backup operation failed for SDDC Manager System Configuration

Products

• VMware SDDC Manager

Issue/Introduction

This error log indicates a couple of issues that are preventing the SDDC Manager daily backup operation from completing successfully. Let's break down the errors and potential solutions:

Error 1: AttributeError: 'NoneType' object has no attribute 'text'

• Location: /var/log/vmware/vcf/sddc-support/backup-/sos.log
• Root Cause: This error suggests that a function in the commandutils.py module is trying to access the text attribute of an object that is actually None. This often happens when a function expecting a response containing text receives a None value instead.
• Possible Solutions:
  • Investigate the get_services_version function: Determine what API calls or operations are being performed within this function and why they might return None. Check for network connectivity issues, API errors, or incorrect API endpoint configurations.
  • Review the export_metadata function: This function appears to be calling get_services_version. Examine the code to understand how the returned value is being used and where it might be becoming None.

Error 2: Unable to create tempDir. java.io.tmpdir is set to /opt/vmware/vcf/lcm/tmp

• Location: /var/log/vmware/vcf/lcm/lcm.log

• Root Cause: The VCF LCM (Life Cycle Manager) component is failing to create a temporary directory because the java.io.tmpdir system environment variable is pointing to /opt/vmware/vcf/lcm/tmp, which is likely either:

  • Insufficient permissions: The VCF LCM process might not have write permissions within the specified directory.
  • Disk space issues: There might not be enough disk space available in the /opt/vmware/vcf/lcm directory to create the temporary directory.

• Possible Solutions:

  • Check Permissions: Ensure that the VCF LCM user or process has write permissions within the /opt/vmware/vcf/lcm/tmp directory. If not, grant the necessary permissions.
  • Verify Disk Space: Examine the disk usage in the /opt/vmware/vcf/lcm directory to confirm that there is sufficient space for creating temporary files. If disk space is low, you might need to clean up files or increase disk space allocation.
  • Alternative Temporary Directory: Consider setting the java.io.tmpdir environment variable to a different, valid temporary directory with sufficient permissions and disk space. However, this might require restarting the VCF LCM component.

Troubleshooting Tips:

• Examine Logs: Review the full log files (/var/log/vmware/vcf/sddc-support/backup-/sos.log, /var/log/vmware/vcf/lcm/lcm.log, and potentially other VCF logs) for more specific error messages and stack traces.
• Check Network Connectivity: Verify that the SDDC Manager has proper network connectivity to the necessary services and API endpoints.
• Verify API Configuration: Confirm that the API endpoints and credentials used by the SDDC Manager backup process are correctly configured.
• Monitor System Resources: Monitor the CPU, memory, and disk usage of the SDDC Manager and other VCF components to rule out resource exhaustion.

By following these steps, you should be able to diagnose and resolve the issues causing the SDDC Manager backup failures.

Environment

VCF 4.5.2

Cause

Incorrect Permissions on the /opt/vmware/vcf/lcm/tmp directory

 

Resolution

1. Take a snapshot of the SDDC Manager VM
2. su to root
3. cd /opt/vmware/vcf/lcm
4. chown vcf_lcm:vcf tmp
5. systemctl restart lcm
6. Retry SDDC Mananger Backup