Multiple VM's are missing from TKGI clusters

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Cluster nodes will be missing including both master and worker VM's.

Using the "tkgi cluster cluster-name" command, verify the actual number of worker nodes. In addition, add the number of the master node. The total number of nodes should be 60 when the "bosh VMs" command is run if worker nodes are 57 and master nodes are 3.

Environment

TKGI -Tanzu Kubernetes Grid Integrated Edition

Cause

It may be due to network issues in the environment, including DHCP issues.

Bosh vm is unable to reach the TKGI cluster VM's


During network problems, if the bosh update-resurection is off

Resolution

Run cck against the cluster if the nodes are missing.

bosh -d service-instance_45rfllrb-4r55-567g-4f45-5ee443a334 cck

root$ bosh -d service-instance_45rfllrb-4r55-567g-4f45-5ee443a334 cck
Using environment '10.10.10.5' as client 'ops_manager'
Using deployment 'service-instance_45rfllrb-4r55-567g-4f45-5ee443a334'
Task 3400456
TaSk 3400456 | 11:23:35 | Scanning 60 VMs: Checking VM states (00:00: 00)
TaSk 3400456 | 11:23:35 | Scanning 60 VMs: 30 OK, 0 unresponsive, 30 missing, 0 unbound (00:00: 00)
Task 3400456 | 11:23:35 | Scanning 60 persistent disks: Looking for inactive disks (00:00:56)
Task 3400456 | 11:25:35 | Scanning 60 persistent disks: 60 OK, 0 missing, 0 inactive, 0 mount-info mismatch (00:00:00)
Task 3400456 Started Wed Jul 30 11:23:35 UTC 2024
Task 3400456 Finished Wed Jul 30 11:25:35 UTC 2024

 Type Description

xxxx missing_vm VM for 'worker/fb3bf62-4566-8997-3456-02ae16g3347t (0)' missing.
xxxx missing_vm VM for 'worker/4rffrt62-2126-4af3-1032-02a5161bf87b (1)' missing.
...
...

30 problems

1: Skip for now
2: Recreate VM without waiting for processes to start
3: Recreate VM and wait for processes to start
4: Delete VM reference
VM for 'worker/fb3bf62-4566-8997-3456-02ae16g3347t (0)' missing (1): 3

For each missing VM we need to select the action plan. Always go with "Recreate VM and wait for processes to start". Select 3.