Fleet Manager LCM or SDDC Manager LCM Tasks appear to be stuck or hung.

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

Symptoms:

Fleet Manager LCM tasks (shown from VCF Ops UI) may appear stuck.

The Fleet LCM parent task does not proceed a single substep for hours, and also is not timing out.

After a VCF deployment workflow successfully completes, specific component deployment tasks or monitoring sub-tasks may appear stuck in an "In-Progress" state in the VCF UI.

Environment

Fleet Manager LCM 9.1.0.0

SDDC Manager LCM 9.1.0.0

Cause

This behavior is typically triggered by transient connectivity issues with internal services during the monitoring phase of a task. Because the VCF Installer is designed with automatic retry logic, the installer often successfully completes the deployment in a subsequent pass by skipping the already-provisioned components. However, the UI state for the initial stuck sub-task may fail to update to "Successful."

Resolution

Validating the deployment

Manually log into the deployed component’s management interface to verify operational health. If it's deployed continue below

 

Fleet Lifecycle Services

• There are two components for the lifecycle services, and each contains two services:
  • Fleet lifecycle:
    • fleetbuild
    • fleetupgrade
  • SDDC lifecycle:
    • sddcbuild
    • sddcupgrade

These services are all running on the VCF services runtime. To find the VCF services runtime each service is running on:

1. From the left pane, select Lifecycle
2. Select VCF Management
3. Select the Task tab
4. Select Fleet lifecycle
5. Find the label Managed by, and select the VCF services runtime.
6. Find the label VCF services runtime FQDN, which should display the FQDN of the VCF services runtime that hosts the Fleet lifecycle.
7. Go through the above steps for the SDDC Lifecycle to find the FQDN of the VCF services runtime that hosts the SDDC lifecycle. If there are multiple VCF Instances, there could be multiple SDDC lifecycle components.

See the screenshots as examples:

Determine Lifecycle Task Stuck

To determine if a lifecycle task is stuck, find the task ID, the FQDN of the VCF services runtime for the Fleet lifecycle (see the steps above), and its administrator credentials. Then run the script as follows. The script will prompt for the administrator password.

./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -t <task_id>

If the task behaves normally, the output will include this:

The task '<task_id>' is normal

If the task is likely stuck in the Fleet lifecycle, the output will include this:

The task '<task_id>' is not making progress for a long period.

You can try to restart the service 'fleetbuild' (or fleetupgrade) on the VCF services runtime '<VCF services runtime FQDN>' to correct this situation.

If the task is likely stuck in the SDDC lifecycle, the output will include this:

The task '<task_id>' is not making progress for a long period.

You can try to restart the service 'sddcbuild' (or sddcupgrade) on the VCF services runtime to correct this situation.
Please identify the VCF services runtime running the 'sddcbuild' (or sddcupgrade)

 

Restart a Service

To restart a service, find the FQDN of the VCF services runtime for the service and its credentials. Then run the script as follows. The script will prompt for the administrator password.

To restart a Fleet lifecycle service, find the FQDN of the VCF services runtime hosting the Fleet lifecycle component. To restart a SDDC lifecycle service, make sure to find the FQDN of the VCF services runtime in the same VCF instance. For example, if the task to deploy a component to vcf-instance-1 is stuck, you will need to find the VCF services runtime in vcf-instance-1.

• To restart fleetbuild
  • ./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r fleetbuild
• To restart fleetupgrade
  • ./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r fleetupgrade
• To restart sddcbuild
  • ./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r sddcbuild
• To restart sddcupgrade
  • ./lcm_service_support.sh -u <admin_user> -s <VCF services runtime FQDN> -r sddcupgrade