SDDC upgrade fails "Setup_Common_Appliance_Platform" in 5.X environment
5.X
Permissions issue for lcm # cd var/log/vmware/vcf/lcm
# cat lcm.err
Traceback (most recent call last):
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/<Bundle_UUID>/thirdparty/cap-platform-setup/bin/upgrade.py", line 33, in <module>
with open('variables.json', 'w') as fp:
PermissionError: [Errno 13] Permission denied: 'variables.json'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/
<Bundle_UUID>
/thirdparty/cap-platform-setup/bin/upgrade.py", line 48, in <module> raise Exception
Exception
As per the error above LCM is unable to create/ write "variables.json"
Also for <Bundle_UUID> see https://knowledge.broadcom.com/external/article/327207/vmware-cloud-foundation-upgrade-bundle.html
Additional logs point to a timeout
LCM logs:# less -i lcm.log
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Upgrade status INPROGRESS for child upgrade: <TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Normalized Parent upgrade status INPROGRESS for parent id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Building normalized user visible upgrade for upgrade id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.i.UpgradeGetterService,http-nio-127.0.0.1-7400-exec-8] Built normalized user visible upgrade for upgrade id: <PARENT TASK ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-8] Upgrade contains sddc_manager [Resource(resourceType=SDDC_MANAGER, resourceId=<TASK ID>, name=<SDDC_FQDN>)]
<Date && Time> ERROR [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-8] Resource type is unknown THIRD_PARTY
<Date && Time> ERROR [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-8] Upgrade error occured: timed out or the process exited without updating status while waiting for upgrade to finish Reference token <TOKEN_ID>
<Date && Time> DEBUG [vcf_lcm,a69649b0a6d433cd,7a3f] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-8] Stage details: Stage(id=6:1, name=null, description=null, type=Setup_Common_Appliance_Platform, order=1, startTime=null, endTime=null, duration=null, status=FAILED, errorType=RECOVERABLE, errorCode=UPGRADE_TIMEDOUT, errorDescription=timed out or the process exited without updating status while waiting for upgrade to finish, remediation=null, resources=null, errors=null, subtaskId=6, subtaskType=UPGRADE, taskId=<TASK ID>, taskType=com.vmware.vcf.lcm.upgrade)
Additional verification, check permission on Folder level for LCM and DOMAINMANAGER [these should be, as the ones highlighted below, anything different can cause an upgrade issue]root@sddc01 [ ~ ]# ls -lahtr /opt/vmware/vcf/
total 124K
drwx------ 2 root root 16K Jun 27 2024 lost+found
-rwxr-xr-x 1 root root 23 Jun 27 2024 version.txt
-rw-r--r-- 1 root root 57K Jun 27 2024 java.default.security
-rw-r--r-- 1 root root 845 Jun 27 2024 java-systemwide-java-options.conf
drwxr-xr-x 9 root root 4.0K Jun 27 2024 commonsvcs
drwxr-xr-x 5 vcf_domainmanager vcf 4.0K Jun 27 2024 domainmanager
drwxr-x--- 4 vcf_lcm vcf 4.0K Jun 27 2024 lcm
drwxr-xr-x 6 root root 4.0K Jun 27 2024 operationsmanager
drwxr-x--x 4 vcf_sddc_manager_ui_app vcf 4.0K Jun 27 2024 sddc-manager-ui-app
drwxr-xr-x 12 root root 4.0K Jun 27 2024 ..
drwxr-xr-x 2 root root 4.0K Jun 27 2024 fips
drwxr-xr-x 2 root root 4.0K Jun 27 2024 OSL
drwxr-xr-x 11 root root 4.0K Jun 27 2024 .
drwxr-x--- 3 vcf_sos vcf 4.0K Nov 29 09:49 sddc-support
In some cases, we've seen permissions being altered - not as per design - i.e. as per above
To resolve the problem for permissions follow these steps
Snapshot of SDDC is recommended.
SSH on the SDDC manager with vcf/ root
Elevate to root if vcf used# su -
#
# cd /opt/vmware/vcf/# chown -R vcf_lcm:vcf lcm
# chown -R vcf_domainmanager:vcf domainmanager
# systemctl restart lcm
# systemctl restart domainmanager
Wait for a few min (e.g 10 min ) for LCM to finish the start sequence, and populate the cache.
Retry sddc manager upgrade