SDDC 5.x upgrade fails with GUI error "Setup_Common_Appliance_Platform"

Products

• VMware SDDC Manager

Issue/Introduction

SDDC upgrade fails "Setup_Common_Appliance_Platform" in 5.X environment 

Environment

5.X 

Cause

Permissions issue for lcm 

# cd var/log/vmware/vcf/lcm
# cat lcm.err
Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/<Bundle_UUID>/thirdparty/cap-platform-setup/bin/upgrade.py", line 33, in <module>
    with open('variables.json', 'w') as fp:
PermissionError: [Errno 13] Permission denied: 'variables.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/<Bundle_UUID>/thirdparty/cap-platform-setup/bin/upgrade.py", line 48, in <module>
    raise Exception
Exception

As per the error above LCM is unable to create/ write "variables.json" 
Also for <Bundle_UUID> see https://knowledge.broadcom.com/external/article/327207/vmware-cloud-foundation-upgrade-bundle.html 

Additional logs point to a timeout 

LCM logs:
# less -i lcm.log 
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Upgrade status INPROGRESS for child upgrade: <TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Normalized Parent upgrade status INPROGRESS for parent id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Building normalized user visible upgrade for upgrade id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Built normalized user visible upgrade for upgrade id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-8] Upgrade contains sddc_manager [Resource(resourceType=SDDC_MANAGER, resourceId=<TASK ID>, name=<SDDC_FQDN>)]
<Date && Time> ERROR [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-8] Resource type is unknown THIRD_PARTY
<Date && Time> ERROR [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-8] Upgrade error occured: timed out or the process exited without updating status while waiting for upgrade to finish Reference token <TOKEN_ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-8] Stage details: Stage(id=6:1, name=null, description=null, type=Setup_Common_Appliance_Platform, order=1, startTime=null, endTime=null, duration=null, status=FAILED, errorType=RECOVERABLE, errorCode=UPGRADE_TIMEDOUT, errorDescription=timed out or the process exited without updating status while waiting for upgrade to finish, remediation=null, resources=null, errors=null, subtaskId=6, subtaskType=UPGRADE, taskId=<TASK ID>, taskType=com.vmware.vcf.lcm.upgrade)

Additional verification, check permission on Folder level for LCM and DOMAINMANAGER [these should be, as the ones highlighted below, anything different can cause an upgrade issue]

root@sddc01 [ ~ ]# ls -lahtr /opt/vmware/vcf/
total 124K
drwx------  2 root                    root  16K Jun 27  2024 lost+found
-rwxr-xr-x  1 root                    root   23 Jun 27  2024 version.txt
-rw-r--r--  1 root                    root  57K Jun 27  2024 java.default.security
-rw-r--r--  1 root                    root  845 Jun 27  2024 java-systemwide-java-options.conf
drwxr-xr-x  9 root                    root 4.0K Jun 27  2024 commonsvcs
drwxr-xr-x  5 vcf_domainmanager       vcf  4.0K Jun 27  2024 domainmanager
drwxr-x---  4 vcf_lcm                 vcf  4.0K Jun 27  2024 lcm
drwxr-xr-x  6 root                    root 4.0K Jun 27  2024 operationsmanager
drwxr-x--x  4 vcf_sddc_manager_ui_app vcf  4.0K Jun 27  2024 sddc-manager-ui-app
drwxr-xr-x 12 root                    root 4.0K Jun 27  2024 ..
drwxr-xr-x  2 root                    root 4.0K Jun 27  2024 fips
drwxr-xr-x  2 root                    root 4.0K Jun 27  2024 OSL
drwxr-xr-x 11 root                    root 4.0K Jun 27  2024 .
drwxr-x---  3 vcf_sos                 vcf  4.0K Nov 29 09:49 sddc-support

In some cases, we've seen permissions being altered - not as per design - i.e. as per above

Resolution

To resolve the problem for permissions follow these steps 
Snapshot of SDDC is recommended.
SSH on the SDDC manager with vcf/ root
Elevate to root if vcf used
# su -
# 
# cd /opt/vmware/vcf/
# chown -R vcf_lcm:vcf lcm
# chown -R vcf_domainmanager:vcf domainmanager
# systemctl restart lcm
# systemctl restart domainmanager

Wait for a few min (e.g 10 min ) for LCM to finish the start sequence, and populate the cache.
Retry sddc manager upgrade