In VCF Operations, the pre-restore Telemetry and Real-Time Metrics components appear as "Unknown" after a restoration, yet the corresponding Namespace and Pod do not exist in Kubernetes.
VMware Cloud Foundation 9.1.x
Telemetry and Real-Time Metrics are exclusively backed up via scheduled backups. In environments where backups were performed solely by manually selecting "BACK UP NOW" in VCF Operations, these components are not included and cannot be restored. Therefore, it is necessary to follow the reinstallation procedure described in this KB article.
The telemetry data appearing as "Unknown" represents pre-restoration telemetry component information held by the restored FLEET LCM and SDDC LCM databases (in the database schema, this corresponds to records with a NULL or empty status). Since these stale components cannot be deleted via the API, the records must be manually removed from the databases.
(1) Log in to the VCF Management Services Control Plane.
Log in to the VCF Operations UI.
Navigate to Build > Lifecycle > VCF Management > Component > VCF services runtime. Under Nodes, identify the IP address of the node where the Node Type is "Control Plane."
SSH into the IP address identified in step 2 using the vmware-system-user account. (In a high-availability setup, connect to any one of the three Control Plane nodes).
Switch to the root user and load the environment variables:
vmware-system-user@vcf-svcruntime-XXXXX [ ~ ]$ sudo -i
[sudo] password for vmware-system-user:
root@vcf-svcruntime-XXXXX [ ~ ]# export KUBECONFIG=/etc/kubernetes/admin.conf
root@vcf-svcruntime-XXXXX [ ~ ]#
(2) Script Execution
1. Download the `fix_unknown_telemetry_1.3.sh` script attached to this KB article and copy it to the Control Plane node (e.g., into the /home/vmware-system-user directory).
2. Grant execute permissions to the script:
chmod +x fix_unknown_telemetry_1.3.sh
3. Execute the script:
./fix_unknown_telemetry_1.3.sh
Note: If multiple candidate records are detected, the script will safely terminate execution to prevent accidental deletion of unintended instance records. If this occurs, do not attempt to modify the database manually; please contact Broadcom Support for assistance.
root@vcf-svcruntime-XXXXX [ /home/vmware-system-user ]# ./fix_unknown_telemetry_1.3.sh
=================================================
1. Identifying Leader Nodes
=================================================
FLEET LCM Leader: vcf-fleet-lcm-db-0
SDDC LCM Leader: vcf-sddc-lcm-db-0
=================================================
2. Fetching stale Telemetry records from Databases
=================================================
Note: The VCF Operations UI displays these components as 'Unknown'.
In the database schema, this corresponds to a NULL or empty status.
Found exactly 1 stale record targeted for deletion:
-------------------------------------------------
[FLEET LCM DB] Table: component
-[ RECORD 1 ]---+-------------------------------------
id | 01a0661b-2045-79f7-840b-############
component_id | 6a654af7-be4a-4e7c-b1cd-############
component_type | TELEMETRY_ACCEPTOR
deployment_type | VSP
fqdn | vcf-instance.example.com
size | small
status |
version | 9.1.0.0.25181946
sddc_lcm_id | d19e7268-d642-4f88-a588-############
[FLEET LCM DB] Table: component_config
-[ RECORD 1 ]+--------------------------------------------------------------------------
component_id | 01a0661b-2045-79f7-840b-############
config_data | {"type":"VspComponentConfig","size":"small","configSpec":{},"volumes":[]}
type | VspComponentConfig
version | 9.1.0.0.25181946
[FLEET LCM DB] Table: upgrade_plan_component
(0 rows)
[SDDC LCM DB] Table: component
-[ RECORD 1 ]---+-------------------------------------
id | 01a06619-0335-742e-a68c-############
component_id | 6a654af7-be4a-4e7c-b1cd-############
component_type | TELEMETRY_ACCEPTOR
deployment_type | VSP
fqdn | vcf-instance.example.com
password |
size | small
username |
version | 9.1.0.0.25181946
-------------------------------------------------
Do you want to permanently delete these records? (yes/no): yes
Confirmation received. Proceeding with deletion...
=================================================
3. Retrieving Database Credentials
=================================================
Credentials retrieved successfully.
=================================================
4. Deleting records from SDDC LCM DB
=================================================
BEGIN
DELETE 1
COMMIT
SDDC LCM DB cleanup done.
=================================================
5. Deleting records from FLEET LCM DB
=================================================
BEGIN
DELETE 0
DELETE 1
DELETE 1
COMMIT
FLEET LCM DB cleanup done.
=================================================
6. Verifying Deletion
=================================================
[FLEET LCM DB] Table: component
Query: SELECT * FROM component WHERE id = '01a0661b-2045-79f7-840b-############';
id | component_id | component_type | deployment_type | fqdn | size | status | version | sddc_lcm_id
----+--------------+----------------+-----------------+------+------+--------+---------+-------------
(0 rows)
[FLEET LCM DB] Table: component_config
Query: SELECT * FROM component_config WHERE component_id = '01a0661b-2045-79f7-840b-############';
component_id | config_data | type | version
--------------+-------------+------+---------
(0 rows)
[FLEET LCM DB] Table: upgrade_plan_component
Query: SELECT * FROM upgrade_plan_component WHERE component_id = '6a654af7-be4a-4e7c-b1cd-############';
plan_id | component_id | component_type | desired_version | eligibility_status | upgrade_dependencies | create_time | update_time | user_input_spec
---------+--------------+----------------+-----------------+--------------------+----------------------+-------------+-------------+-----------------
(0 rows)
[SDDC LCM DB] Table: component
Query: SELECT * FROM component WHERE component_id = '6a654af7-be4a-4e7c-b1cd-############';
id | component_id | component_type | deployment_type | fqdn | password | size | username | version
----+--------------+----------------+-----------------+------+----------+------+----------+---------
(0 rows)
Verification passed: All target records have been successfully deleted from both databases (0 rows remaining).
=================================================
Cleanup completed successfully!
Please proceed with reinstalling Telemetry via API.
=================================================
(3) Verification
Log in to the VCF Operations UI and verify that the "Unknown" Telemetry component has been successfully removed from the UI.
(1) Reinstalling Telemetry
1. Define the required variables (Replace <FLEET_FQDN> and <[email protected]_password> with your actual values):
export VCFMS_FC='<FLEET_FQDN>'
export VCFMS_FC_PASSWORD='<[email protected]_password>'
VCFMS_FC_TOKEN=$(curl -ks --request POST \
--url "https://$VCFMS_FC/api/v1/identity/token" \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data grant_type=password \
--data "[email protected]" \
--data "password=$VCFMS_FC_PASSWORD" \
| jq -r '.access_token')
# Verify the token is acquired successfully
echo $VCFMS_FC_TOKEN
2. Obtain the sddcLcmId for the affected instance from VCF Management Services (Replace <INSTANCE_FQDN> with the FQDN of the affected VCF instance):
export INSTANCE_FQDN='<INSTANCE_FQDN>'
sddcLcmId=$(curl -ks "https://${VCFMS_FC}/fleet-lcm/v1/components?includeConsumptionVsp=true&includeVcdMigrator=true" \
-H "Authorization: Bearer $VCFMS_FC_TOKEN" | \
jq -r --arg fqdn "$INSTANCE_FQDN" '.components[] | select(.componentTypeDescription == "SDDC lifecycle" and .fqdn == $fqdn) | .id')
# Verify the result (Ensure only one ID is returned)
echo $sddcLcmId
3. Reinstall Telemetry. (Note: Ensure the version and size parameters in the JSON payload match your original configuration before executing the command).
curl -ks -X POST "https://$VCFMS_FC/fleet-lcm/v1/components" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $VCFMS_FC_TOKEN" \
-d '{
"componentSpecs": [
{
"sddcLcmId": "'"${sddcLcmId}"'",
"deploymentType": "VspComponentSpec",
"componentType": "TELEMETRY_ACCEPTOR",
"version": "9.1.0.0",
"configSpec": {
"size": "small"
}
}
]
}' | jq
4. Monitor the installation task in the VCF Operations UI: Navigate to Build > Lifecycle > VCF Management > Tasks and verify that the task Install components: Telemetry SDDC LCM completes successfully.
(2) Deleting and Reinstalling Real-time Metrics Components
(Note: If you are performing this immediately after reinstalling Telemetry in the same terminal session, your VCFMS_FC_TOKEN is already active and you can skip Step 1).
1. Define the required variables (Replace <FLEET_FQDN> and <[email protected]_password> with your actual values):
export VCFMS_FC='<FLEET_FQDN>'
export VCFMS_FC_PASSWORD='<[email protected]_password>'
VCFMS_FC_TOKEN=$(curl -ks --request POST \
--url "https://$VCFMS_FC/api/v1/identity/token" \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data grant_type=password \
--data "[email protected]" \
--data "password=$VCFMS_FC_PASSWORD" \
| jq -r '.access_token')
# Verify the token is acquired successfully
echo $VCFMS_FC_TOKEN
2. Retrieve the component IDs for the Real-time Metrics and the Real-time Metrics Store:
curl -k -s "https://$VCFMS_FC/fleet-lcm/v1/components?includeConsumptionVsp=true&includeVcdMigrator=true" \
-H "Authorization: Bearer $VCFMS_FC_TOKEN" | \
jq '.components[] |
select(.componentTypeDescription == "Real-time metrics" or .componentTypeDescription == "Real-time metrics store") |
{id, componentTypeDescription}'
3. Delete the components using the IDs obtained in step 2:
# component id should match the id for real-time metrics from step 2
curl -X DELETE "https://$VCFMS_FC/fleet-lcm/v1/components/<real-time metrics component id step2>" \
-k -H "Authorization: Bearer $VCFMS_FC_TOKEN" | jq
# component id should match the id for real-time metrics store from step 2
curl -X DELETE "https://$VCFMS_FC/fleet-lcm/v1/components/<real-time metrics store component id step2>" \
-k -H "Authorization: Bearer $VCFMS_FC_TOKEN" | jq
4. Monitor the deletion tasks in the VCF Operations UI: Navigate to Build > Lifecycle > VCF Management > Tasks and verify that the tasks Delete Real-time metrics component and Delete Real-time metrics store component complete successfully.
5. Reinstall the Real-time Metrics components: Once the deletion tasks are complete, install the real-time metrics components using the Add Component user interface menu in VCF Operations.