Management Cluster Create / Upgrade fails as it cannot pull artifacts from a decommissioned artifactory
search cancel

Management Cluster Create / Upgrade fails as it cannot pull artifacts from a decommissioned artifactory

book

Article ID: 371486

calendar_today

Updated On:

Products

VMware Telco Cloud Automation

Issue/Introduction

This document details an issue specific to the Telco Cloud Automation (TCA) management cluster create/upgrade process getting stuck and eventually timing out.

  • New Management Cluster Creation: When creating a new cluster, the status in TCA remains stuck at “Posted request to deploy cluster” and eventually fails with the error “cluster creation failed,” prompting you to check logs for more details about the failure.
  • Management Cluster Update or Upgrade: If the upgrade is triggered from TCA, it remains in the upgrading state for an extended period before timing out. However, if the upgrade is initiated directly by the control-plane, an additional node spins up, but no further progress is made. Despite the upgrade being stuck, the cluster remains functional, and all LCM processes continue to work.

The Kubelet logs (`journalctl -u kubelet`) and containerd logs (`journalctl -u containerd`) on the new node will display log entries similar to the following:

Jun 29 04:30:36 xxxxxxxxxxxxxx kubelet 1225: 20629 04:30:36.9091521225 remote runtime.go:201] "RunPodSandbox from runtime service failed" ere.com tka pause:3.7": failed to com/tka/pause: 3.7\": pulling from host projects-stg.registry.vmware.com failed with status code [manifests 3.71: 503 Service Unavailable"

Environment

2.3.x, 3.x

Cause

During the LCM process of the management cluster, a new node is created and attempts to pull necessary artifacts from remote internet based artifacts. One of the repositories, "projects-stg.registry.vmware.com," was decommissioned by Broadcom as of July 2024 since it was intended for internal use only.

There is a public equivalent registry for external access, "project.registry.vmware.com," which should be used. However, the BYOI templates that were released as part of the Telco Cloud Automation release contain hardcoded configurations to access "projects-stg.registry.vmware.com." Consequently, attempts to access the decommissioned URL will result in a 503 service unavailable error.

Resolution

Resolution

  1. Download the BYOI template from and stage it in a location accessible to vSphere (vCenter).
    1. Browse to Broadcom Support
    2. Login using your Broadcom credentials
    3. Ensure that you select the Software Defined Edge group in the drop down on the top right
    4. Browse to My Downloads > VMware Telco Cloud Automation
    5. Expand the VMware Telco Cloud Automation row by clicking on it
    6. Select the appropriate release number
    7. Select the Drivers & Tools tab
    8. Read through and agree to the Broadcom Terms and Conditions.
    9. Find the BYOI template and click on the download icon.
    10. Download the latest BYOI template OVAs that your management and workload clusters run on from the OS and Kubernetes version line
  2. Login to the vCenter Web interface.
  3. Right-click on the datacenter and select "Deploy OVF Template". Note: same location with the original template 

     4. Select the downloaded BYOI template file and click "Next".

    5. Complete the wizard and wait for the template upload to finish.

    6. Right-click on the imported virtual machine, then select Template → Convert to Template.

 

   7. Note the name of the current template in use, for example: photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-21438772.

   8. Rename the existing template, for example, rename photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-21438772 to photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-21438772.old.

   9. Rename the newly imported template to match the same old name, for example, rename. photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-<newbuild> to photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-21438772.

  10. Repeat steps 1 through 9 for all affected templates.

Affected BYOI template version

TCA Version

TKG Version

Kubernetes Version

New Template

2.3

2.3.0.1

2.3.0.2

3.1

3.1.1

2.1.1

1.24.10

photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-24058474

3.0

3.1

3.1.1

2.2.0

1.25.7

photon-3-kube-v1.25.7-vmware.2-tkg.1-8795de-24071639

3.0

3.1

3.1.1

2.3.1

1.26.8

photon-3-kube-v1.26.8-vmware.1-tkg.1-ff40cd-24072200

3.1

3.1.1

2.4.1

1.27.5

photon-3-kube-v1.27.5-vmware.1-tkg.1-cac282-24071617

 

 

Additional Information

This issue is not applicable for air gapped deployments

Powered by Wolken Software