Validating file or directory permissions before performing the upgrade in VMware Cloud Foundation

Products

• VMware Cloud Foundation

Issue/Introduction

This is to highlight permission issues that can potentially cause issues during the VMware Cloud Foundation upgrades.

Symptoms:

• Bundle downloads in online mode fails. 
• Sddc manager upgrade fails due to incorrect file/directory permissions or ownership.

Environment

VMware Cloud Foundation 4.4
VMware Cloud Foundation 4.2.x
Vmware Cloud Foundation 4.5
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 4.0.x

Cause

Permissions or ownerships of files/directories used by SDDC manager services are incorrect.

Resolution

Permissions or ownerships have to be fixed in the environment to avoid issues.

Workaround:
Run the attached sddcmanager_fs_permission_checks.py file to validate files/directories permission and ownership.

Step 1: Login to SDDC Manager VM via SSH and switch to root

Step 2: Copy the attached python script file (sddcmanager_fs_permission_checks.py) via SCP to SDDC Manager under /home/vcf

Step 3: Set permissions for script file
    cd /home/vcf
  chown root:root sddcmanager_fs_permission_checks.py
  chmod 750 sddcmanager_fs_permission_checks.py

Prior to executing the script, ensure there are no incomplete log bundles leftover from sos tool.
Execute command: `ls -lah /opt/vmware/vcf/sddc-support`
If there are any directories starting with sddc-*, then these need to be removed.
Execute command: `rm -rf /opt/vmware/vcf/sddc-support/sddc-*`

Step 4: Run the python script file
    python sddcmanager_fs_permission_checks.py

4a.If the script fails with a stacktrace similar to the following:

Traceback (most recent call last):
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 383, in <module>
main()
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 377, in main
checker.inspect_directories()
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 312, in inspect_directories
self.inspect_directory(path, owner)
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 307, in inspect_directory
os.path.join(dirpath, d_name), dir_owner)
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 307, in inspect_directory
os.path.join(dirpath, d_name), dir_owner)
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 307, in inspect_directory
os.path.join(dirpath, d_name), dir_owner)
[Previous line repeated 1 more time]
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 296, in inspect_directory
file_owner = self.__validate(f_path, root_owner)
File "/opt/vmware/vcf/lcm/lcm-app/bin/sddcmanager_fs_permission_checks.py.copy", line 206, in __validate
status = os.stat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/vmware/vcf/sddc-support/sddc-202301100925/var/log/dracut.log'

Then there is an invalid symlinked file which is causing the script to exit prematurely.
To move past this, manually create the file it is complaining about with 750 permissions:
Example command based on above stacktrace: `touch /opt/vmware/vcf/sddc-support/sddc-202301100925/var/log/dracut.log && chmod 750 /opt/vmware/vcf/sddc-support/sddc-202301100925/var/log/dracut.log`

After running the above, retry the script (i.e. retry step 4)

Step 5: Check for errors reported on the console
    Example:-
    Permission Error :  
{'path': '/opt/vmware/vcf/commonsvcs/scripts/fips/init-fips.sh', 'message': 'owner permission bits are "r-x". Expected at the minimum "rw-"', 'remediation': 'Please make sure files under directory owned by a specific non root owner have both read and write permission'}

     Ownership Error :  {'path': '/var/log/vmware/vcf/lcm', 'message': 'owner is root. Expected vcf_lcm', 'remediation': 'Please make sure files under directory owned by a specific owner have the same owner unless the parent directory is owned by "root" user'}

Step 6: Based on remediation, set correct permission and ownership   
     Set permission example:- 
          chmod 750 /opt/vmware/vcf/commonsvcs/scripts/fips/init-fips.sh

    Set ownership example:- 
          chown vcf_lcm:vcf /var/log/vmware/vcf/lcm 
     

If above fails to provide ownership to child directories within the parent directory, then run it recursively. For example:- 

chown -R vcf_lcm:vcf /opt/vmware/vcf/kcm/lcm-tools/bin

Step 7: Rerun the script (Step-4) to verify that all file/directory permission and ownership issues are resolved.