Fleet Manager LCM or SDDC Manager LCM Tasks appear to be stuck or hung.

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

Fleet Manager LCM tasks (shown from VCF Ops UI) may appear stuck.
The Fleet LCM parent task does not proceed a single substep for hours, and also is not timing out.
After a VCF deployment workflow successfully completes, specific component deployment tasks or monitoring sub-tasks may appear stuck in an "In-Progress" state in the VCF UI.

Environment

Fleet Lifecycle 9.1
SDDC Lifecycle 9.1
VCF Operations 9.1

Cause

This behavior is typically triggered by transient connectivity issues with internal services during the monitoring phase of a task. Because the VCF Installer is designed with automatic retry logic, the installer often successfully completes the deployment in a subsequent pass by skipping the already-provisioned components. However, the UI state for the initial stuck sub-task may fail to update to "Successful."

Resolution

Validating the deployment

Manually log into the deployed component’s management interface to verify operational health. If it's deployed continue below

Fleet Lifecycle Services

There are two components for the lifecycle services, and each contains two services:
- Fleet lifecycle:
  - fleetbuild
  - fleetupgrade
- SDDC lifecycle:
  - sddcbuild
  - sddcupgrade

These services are all running on the VCF services runtime. To find the VCF services runtime each service is running on:

From the left pane, select Lifecycle
Select VCF Management
Select the Task tab
Select Fleet lifecycle
Find the label Managed by, and select the VCF services runtime.
Find the label VCF services runtime FQDN, which should display the FQDN of the VCF services runtime that hosts the Fleet lifecycle.
Go through the above steps for the SDDC Lifecycle to find the FQDN of the VCF services runtime that hosts the SDDC lifecycle. If there are multiple VCF Instances, there could be multiple SDDC lifecycle components.

See the screenshots as examples:

Determine Lifecycle Task Stuck

To determine if a lifecycle task is stuck, find the task ID, the FQDN of the VCF services runtime for the Fleet lifecycle (see the steps above), and its administrator credentials. Then run the attached script on the SDDC manager as follows. The script will prompt for the administrator password. The account needed is the [email protected]l account

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -t <task_id>

If the task is behaving normally, the output will include the following:
```
The task '<task_id>' is normal
```
If the task is likely stuck in the Fleet lifecycle, the output will include this:
```
The task '<task_id>' is not making progress for a long period. 
```
If necessary, restart the service 'fleetbuild' (or fleetupgrade) on the VCF services runtime to correct this situation.
If the task is likely stuck in the SDDC lifecycle, the output will include the following:
```
The task '<task_id>' is not making progress for a long period. 
```
If necessary restart the service 'sddcbuild' (or sddcupgrade) on the VCF services runtime to correct this situation.

Restart a Service

To restart a service, find the FQDN of the VCF services runtime for the service and its credentials. Then run the script as follows. The script will prompt for the administrator password.

To restart a Fleet lifecycle service, find the FQDN of the VCF services runtime hosting the Fleet lifecycle component. To restart a SDDC lifecycle service, make sure to find the FQDN of the VCF services runtime in the same VCF instance. For example, if the task to deploy a component to vcf-instance-1 is stuck, it is necessary to find the VCF services runtime in vcf-instance-1.

To restart fleetbuild:

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r fleetbuild

To restart fleetupgrade:

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r fleetupgrade

To restart sddcbuild:

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r sddcbuild

To restart sddcupgrade:

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r sddcupgrade

Additional Information

<admin_user> is the [email protected] user

At deployment, this user has the same password as vmware-system-user but may differ afterwards.

If necessary, this password can be reset using the following KB:
Resetting the [email protected] password for VMware Cloud Foundation Services Runtime

Attachments

lcm_service_support.sh get_app