SDDC Manager upgrade fails with ERROR: CAP services are not enabled in SDDC Manager
search cancel

SDDC Manager upgrade fails with ERROR: CAP services are not enabled in SDDC Manager

book

Article ID: 376799

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • SDDC Manager upgrade fails at Set up common appliance platform with error 
    CAP services are not enabled in SDDC Manager

     

  • cap-workflow-engine.service shows as failed when checked with: 
    systemctl status cap-workflow-engine.service

     

  • Error in /var/log/vmware/vcf/lcm/thirdparty/upgrades/17de####-####-####-####-########5564/vcf-platform/cap-platform-setup/workflow.log 
    INFO: http://127.0.0.1:15051/capengine/api/v1/workflows is not accessible, retry after 10 seconds
INFO: URL: http://127.0.0.1:15051/capengine/api/v1/workflows
ERROR: RC: , OUT:  ERR: HTTPConnectionPool(host='127.0.0.1', port=15051): Max retries exceeded with url: /capengine/api/v1/workflows (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x######10>: Failed to establish a new connection: [Errno 111] Connection refused')) NoneType: None
INFO: http://127.0.0.1:15051/capengine/api/v1/workflows is not accessible, retry after 10 seconds
NFO: URL: http://127.0.0.1:15051/capengine/api/v1/workflows
ERROR: RC: , OUT:  ERR: HTTPConnectionPool(host='127.0.0.1', port=15051): Max retries exceeded with url: /capengine/api/v1/workflows (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x######10>: Failed to establish a new connection: [Errno 111] Connection refused')) NoneType: None
NFO: http://127.0.0.1:15051/capengine/api/v1/workflows is not accessible, retry after 10 seconds
INFO: URL: http://127.0.0.1:15051/capengine/api/v1/workflows
ERROR: RC: , OUT:  ERR: HTTPConnectionPool(host='127.0.0.1', port=15051): Max retries exceeded with url: /capengine/api/v1/workflows (Caused by NewConnectionError('<urllib3.connect with data OrderedDict([('upgradeId', 'f2e3####-####-####-####-########3631'), ('resourceId', ''), ('upgradeStatusCode', 'COMPLETED_WITH_FAILURE'), ('progress', 0), ('error', OrderedDict([('errorCode', 2), ('errorDescription', 'http://127.0.0.1:15051/capengine/api/v1/workflows is not accessible')])), ('startTime', 1724776678), ('endTime', 1724777010)])
yyy-mm-dd hh-mm-ss,154: ERROR:
Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/cap_platform_setup.py.copy", line 362, in <module> cap_upgraded, cap_header = wrapper.upgrade_cap(CAP_PLATFORM_SETUP_LIBRARY_PATH)
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 574, in upgrade_cap  return self.is_cap_service_running(cap_header), cap_header
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 319, in is_cap_service_running errmsg=error_message)
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 187, in update_status  raise Exception
Exception

INFO: URL: http://localhost/lcm/about

INFO: Updated /var/log/vmware/vcf/lcm/thirdparty/upgrades/17de####-####-####-####-########5564/vcf-platform/cap-platform-setup/cap_platform_setup.status status file with data OrderedDict([('upgradeId', 'f2e3####-####-####-####-########3631'), ('resourceId', '8fb9####-####-####-####-########fd26'), ('upgradeStatusCode', 'COMPLETED_WITH_FAILURE'), ('progress', 0), ('error', OrderedDict([('errorCode', 2), ('errorDescription', 'http://127.0.0.1:15051/capengine/api/v1/workflows is not accessible')])), ('startTime', 1724776678), ('endTime', 1724777010)])
ERROR: CAP services are not enabled in SDDC Manager
INFO:
INFO: RC: 1, OUT:
INFO: ERR: Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/cap_platform_setup.py.copy", line 362, in <module>   cap_upgraded, cap_header = wrapper.upgrade_cap(CAP_PLATFORM_SETUP_LIBRARY_PATH)
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 574, in upgrade_cap  return self.is_cap_service_running(cap_header), cap_header
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 319, in is_cap_service_running  errmsg=error_message)
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/1836####-####-####-####-########72d9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 187, in update_status  raise Exception

     

  • Error in /var/log/vmware/capengine/core-engine.log 
    workflowconfig.go:286: Workflow JSON Object: &{Name:cap-update-revert ExecOrder:[snapshot-check lvm-revert revert-update-changes file-sync-service-setup] TaskExtensionPath: RebootRequired:false TaskList:[{Name:snapshot-check PluginPath:/usr/lib/vmware-capengine/coreplugins/snapshotcheck.so ScriptPath: ScriptArgs:[] IsResumable:false MaxRetryCount:3 ErrorHandlerName: IsReserved:true IsExtensionTask:false} {Name:lvm-revert PluginPath:/usr/lib/vmware-capengine/coreplugins/lvmrevert.so ScriptPath: ScriptArgs:[] IsResumable:false MaxRetryCount:3 ErrorHandlerName: IsReserved:true IsExtensionTask:false} {Name:revert-update-changes PluginPath:/usr/lib/vmware-capengine/coreplugins/revertupdate.so ScriptPath: ScriptArgs:[] IsResumable:false MaxRetryCount:3 ErrorHandlerName: IsReserved:true IsExtensionTask:false} {Name:file-sync-service-setup PluginPath:/usr/lib/vmware-capengine/coreplugins/root_file_sync.so ScriptPath: ScriptArgs:[] IsResumable:false MaxRetryCount:3 ErrorHandlerName: IsReserved:true IsExtensionTask:false}] ErrorHandlers:map[]}
database.go:464: Executing Query: [SELECT _id, workflowId, name, taskOrder, createdTime, lastModifiedTime, isTaskDeleted FROM ERROR_HANDLER WHERE workflowId = ?] with parameters [%!s(int=10)]
main.go:24: Invalid workflows found.
main.go:25: Failed to start Common Appliance Platform Workflow Engine

 

Environment

VMware Cloud Foundation 4.5.1

Cause

When the workflow is altered , the new binary path(s) is not consistent with path referred in altered workflow. Due to which, it would fail to start cap workflow engine.

Resolution

  1. Take the snapshot of the SDDC VM
  2. SSH into the SDDC manager VM using VCF as user name and switch to root
  3. Remove the workflow database 
    rm -rf /storage/lifecycle/capengine/workflow.db
  4. Check files installed by CAP services on SDDC Manager 
    rpm -ql Vmware-capengine
     
  5. Uninstall the Vmware-capengine and Vmware-capupdate RPMs to remove the corrupted workflow definition 
    rm -rf /usr/lib/vmware-capengine/
rm -rf /usr/lib/vmware-capupdate/
tdnf  --disablerepo=* remove Vmware-capengine Vmware-capupdate
rm -rf /etc/vmware/cap/
rm -rf /usr/lib/vmware-capengine/
  6. Install the same RPMs again - Vmware-capengine and Vmware-capupdate
    • Location for the RPMs - /var/log/vmware/vcf/lcm/thirdparty/bundles/<bundle_id>/thirdparty/cap-platform-setup/lib 
      rpm -i Vmware-capengine-1.0.0.3-10001489.x86_64.rpm
rpm -i Vmware-capupdate-1.0.0.3-10001489.x86_64.rpm
  7. Start the Cap services 
    systemctl start cap-workflow-engine.service
  8. Confirm the cap services are running 
    systemctl status cap-workflow-engine.service
  9. Re-try the upgrade
Powered by Wolken Software