Certificate replacement task for Log Management fails with error "[VCFMS-MACHINE-003]" in VMware Cloud Foundation Operations

search cancel

Certificate replacement task for Log Management fails with error "[VCFMS-MACHINE-003]" in VMware Cloud Foundation Operations

book

Article ID: 452681

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

In VMware Cloud Foundation (VCF) Operations, the certificate replacement task for the Log Management component with the following error:

Error Code: vmsp.cert.replacement.error
vmsp.cert.replacement.error
Certificate task REPLACE_CERTIFICATE for <FQDN> has failed. Error message: Certificate replacement error - VMSP Task status with task ID <ID> is Failed. Details: ERROR: Internal error occurred. Please retry the operation or contact support if the issue persists. [VCFMS-MACHINE-003]

Following errors are logged in VCF Operations /storage/log/vcops/log:

#component-manager-plugin-<ID>.log

INFO component-manager-plugin 32025 [ops@#### threadId="2561" threadName="ComponentManagerTaskEngine-4" operationId=""] [com.vmware.vrealize.lcm.platform.automata.service.Task.injectFailedEventWithLocale] - Injecting task failure event. Error Code : 'vmsp.cert.replacement.error', Retry : 'false', Causing Properties : '{ CAUSE ::  }'
com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.CertificateGenericException: Certificate task REPLACE_CERTIFICATE for <FQDN> has failed.  Error message: Certificate replacement error - VMSP Task status with task ID <ID> is Failed. Details: ERROR: Internal error occurred. Please retry the operation or contact support if the issue persists. [VCFMS-MACHINE-003]
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.CertTaskUtil.handleCertificateTaskFailure(CertTaskUtil.java:177)
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.CertTaskUtil.handleTerminatedTask(CertTaskUtil.java:142)
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.CertTaskUtil.monitorTaskInternal(CertTaskUtil.java:106)
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.CertTaskUtil.monitorCertificateTask(CertTaskUtil.java:82)
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.CertTaskUtil.monitorCertificateTask(CertTaskUtil.java:66)
        at com.vmware.vrealize.lcm.plugin.core.vrops.certificatemanagement.tasks.ReplaceCertificateTask.execute(ReplaceCertificateTask.java:48)
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:50)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

#vcops-bridge-<ID>.log

INFO vcfops-bridge 32025 [ops@#### threadId="2561" threadName="ComponentManagerTaskEngine-4" operationId=""] [com.vmware.vcops.bridge.server.vcf.certificate.task.VcfCertificateTaskHelper.getCertManageTask] - getCertificateTask from KV store : Task{id='<ID>', name=REPLACE_CERTIFICATE, orchestratorType=SDDC, orchestratorName='', status=FAILED, creationTimestamp=##########, lastUpdatedOn=##########, additionalInfo={CERTIFICATE_MANAGEMENT_TASK_ID=<TASK_ID>, resource_key=<ID>}, orchestratorTask=OrchestratorTask{id='<ID>', type='null', name='null', description='null', status=FAILED, lastUpdatedOn='null', errors=[Error{message='Certificate replacement error - VMSP Task status with task ID <ID> is Failed. Details: ERROR: Internal error occurred. Please retry the operation or contact support if the issue persists. [VCFMS-MACHINE-003]', code='vmsp.cert.replacement.error'}]}}

Environment

VCF Operations 9.1
VCF Services Runtime 9.1
VCF Log Management 9.1

Cause

The certificate replacement failure occurs because the underlying configure-component workflow fails during the check-machine-rollout and validate-machine-rollout stages with an Error (exit code 1). This indicates that the Kubernetes deployment or StatefulSet for Log Management in the ops-logs namespace did not reach a ready state within the expected timeout.

One of the machines is stuck in Deleting phase in VCF Services Runtime Cluster:

k get machines -A

Upon investigating the VCF Services Runtime cluster state and logs, this timeout and machine in Deleting phase is driven by two underlying issues:

vCenter Thumbprint Mismatch: The system is unable to authenticate and create a vCenter session because the vCenter certificate thumbprint cached in the VMSP platform no longer matches the actual vCenter thumbprint.

#k get cluster <cluster> -n vmsp-platform -o yaml

message: '* VCenterAvailable: failed to create vCenter session: failed to create client: Post "https://<VC-FQDN>/sdk": host "<VC-FQDN>:443" thumbprint does not match "<THUMBPRINT>"'
reason: NotReady
status: "False"
type: InfrastructureReady

Stale Machine in Deleting State: A synthetic health check fails because one of the worker nodes/machines is stuck in a Deleting state for an extended period, preventing the rollout from succeeding.

#k get task configure-component-<ID> -n vmsp-platform -o yaml

status:
  cancellable: true
  currentStageName: config-component-init
  messages:
  - default: Internal error occurred. Please retry the operation or contact support
      if the issue persists. [VCFMS-MACHINE-003]
    id: com.vmware.vcfms.system.GenericError
    level: ERROR
    messageCode: VCFMS-MACHINE-003
    stage: config-component-init
...
  stages:
  - description: Configuration Initialization for VCF Component
    errors:
    - 'configure-component-<ID>: retryStrategy.expression
      evaluated to false'
    - 'configure-component-<ID>(0): Failed'
    - 'component-worker-node-resize(0): Failed'
    - 'component-worker-node-resize: retryStrategy.expression evaluated to false'
    - 'validate-machine-rollout(0): Failed'
    - 'validate-machine-rollout: retryStrategy.expression evaluated to false'
    - 'check-machine-rollout: retryStrategy.expression evaluated to false'
    - 'check-machine-rollout(0): Error (exit code 1)'
    name: config-component-init
    stageType: Workflow
    status: Failed

Note:

Run the above commands on Control Plane of VCF Services Runtime cluster.
To get the info of cluster, run any of the following commands:
```
k get clusters -A
k get machines -A
```
To check the specific output in cause (2), you need to trigger the certificate replacement task in VCF Operations UI and query the VMSP Task custom resource in VCF Services Runtime cluster and output it in YAML format. Because the task name contains a randomly generated tracking ID, you should first list the recent tasks in the vmsp-platform namespace to grab the exact name:
```
k get tasks -n vmsp-platform
```

Resolution

To resolve this issue, you must manually extract the new vCenter certificate and thumbprint, patch the VMSP platform configurations, and restart the vSphere CSI pods so they can successfully authenticate.

Follow the steps below from the Control Plane of VCF Services Runtime Cluster:

Retrieve the new certificate from vCenter:(Replace VC-Host with the actual vCenter FQDN)

vcenterCertificate=$(
    echo | openssl s_client -showcerts -connect <VC-Host>:443 2> /dev/null |
    sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | base64 | tr -d '\n'
)

Echo vcenterCertificate:
```
echo $vcenterCertificate
```

Extract the thumbprint from the retrieved certificate:

thumbprint=$(
    echo $vcenterCertificate | base64 -d | openssl x509 -noout -fingerprint -sha1 | cut -d= -f2
)

Patch the vmsp-platform PackageDeployment with the new certificate and thumbprint:

k patch pd vmsp-platform -n vmsp-platform --type=merge --patch-file /dev/stdin <<EOF
{
  "spec": {
    "values": {
      "provider": {
        "vsphere": {
          "thumbprint": "${thumbprint}",
          "tlsCertificate": "${vcenterCertificate}"
        }
      }
    }
  }
}
EOF

Patch the platform-trust secret with the new certificate:

k patch secret platform-trust -n vmsp-platform --type=merge \
  -p "$(jq -n --arg cert "$vcenterCertificate" '{data: {"ca.crt": $cert}}')"

Restart the vsphere-csi-node and vsphere-csi-controller pods. Identify the pod IDs in the kube-system namespace and delete them to force a recreation:
```
k get pods -n kube-system
k delete pods vsphere-csi-node-<id> -n kube-system
k delete pods vsphere-csi-controller-<id> -n kube-system
```

Wait for the PackageDeployment to become ready:

k wait packagedeployment/vmsp-platform -n vmsp-platform --for=condition=Ready

Perform a final rollout restart of the CSI controller: Note: Restarting this service is non-impacting for workloads and ensures the storage system is immediately remediated following the certificate update.
```
k rollout restart deployments/vsphere-csi-controller -n kube-system
```
Verify if there are any stale machines in a Deleting phase blocking the rollout by running:
```
k get machines -A
```
Retry the Certificate replacement task in VCF Operations UI. It should now complete successfully.

Feedback

thumb_up Yes

thumb_down No