How to troubleshoot a corrupt ssh-key configuration after Tanzu Kubernetes Grid cluster creation

Products

• Tanzu Kubernetes Grid

Issue/Introduction

There can be situations when the SSH key is corrupt and you are not able to access the specific node.

For example, if you try to SSH to the management control plane during post deployment, it asks for a password which was never specified.

This is because the SSH key was not passed correctly during installation or the SSH key is corrupt.

Symptoms:
When trying to SSH to the management control plane or any node in Tanzu Kubernetes Grid (TKG), it prompts for a password.

Resolution

There is no resolution to resolve this issue. The workaround explains how to make sure you are passing the correct SSH key.

Workaround:
Depending on where you are trying to SSH, such as the the management control plane, you will have to follow the below steps to add the correct SSH key.

1. Shutdown or force stop the management control plane VM from the vCenter GUI and remove the disk. This will be the OS disk, and usually it's 40GB.

Note: Please don't check the option of "Delete files from datastore".

To remove the disk post shutting down the VM from the vCenter GUI, follow these steps:

a. Select Action

b. Edit the VM settings.

c. Choose the disk that we need to removed. Make a note of the name.

d. Hover on the right side and click on the (x) to remove.


2. Add this disk to a different VM, for example a jumpbox, where you can log in.
On the VM from vCenter GUI, follow these steps: 

a. Select Action.

b. Edit the VM settings.

c. Add a new device.

d. Select the existing hard disk. Browse the VM folder name and add the correct disk.


3. Once the disk is added to the jumpbox VM and you have the correct SSH key, follow these steps.

a. To mount this disk, first run "sudo fdisk -l". You will get an output similar to the one below, and the last disk is usually the one added. Check for the size.

Partition table entries are not in disk order.

Disk /dev/sdb: 40 GiB, 42949672960 bytes, 83886080 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0C603E73-508C-4481-8514-8C893DD57697

Device     Start      End  Sectors Size Type
/dev/sdb1   2048    10239     8192   4M BIOS boot
/dev/sdb2  10240    30719    20480  10M EFI System
/dev/sdb3  30720 83886046 83855327  40G Linux filesystem
================================================================

b. Create a directory to mount "/dev/sdb<>"  using  "mkdir control-plane".

c. Run the mount command:

sudo mount /dev/sdb3 /home/ubuntu/control-plane/

d. Copy contents in the VM from path "~/.ssh/id_rsa.pub" to the "control-plane/home/capv/.ssh/authorized_keys"

e. Next unmount the disk from the jumpbox:

sudo umount /control-plane

4. The last step is to remove the disk from the jumpbox and re-add it to the management control plane VM.

Note: Please don't check the option, "Delete files from datastore".

On the jumpbox VM from the vCenter GUI, follow these steps:

a. Select Action.

b. Edit the VM settings.

c. Choose the disk that you need to remove. Make a note of the name.

d. Hover on the right side and click on the (x) to remove.

To add the disk back to the control-plane VM, follow these steps:

a. Select Action.

b. Edit the VM settings.

c. Add a new device.

d. Select the existing Hard disk. Browse the VM folder name and add the correct disk.


5. Power-On the management control plane VM and try to SSH from the jumpbox VM.

For example:

ssh capv@<IP>