SSH access to VKS Cluster Node in NSX-VPC environment

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In VKS NSX-VPC environments,  the official troubleshooting procedure may fail due to some networking issues that prevent container image downloads.

For troubleshooting purposes, this KB provides an alternative method to SSH into target VKS nodes using the Avi Load Balancer.

Environment

vSphere Kubernetes Service in NSX-VPC

Resolution

Create a Virtual Service in AVI for SSH access

1. Open the AVI Controller Web UI.
2. Switch the Tenant view in the top-right corner, then navigate to Applications --> Virtual Services --> CREATE VIRTUAL SERVICE --> Basic Setup.
3. Select Cloud: nsx-cloud --> Next
4. Select VRF Context: "Select target vSphere Namespace VRF" --> Next
5. Create a "New Virtual Service"
   1. Name: test-ssh
   2. Application Type: L4
   3. Service: 22
   4. VS VIP: Select Create VS VIP
      1. VIPs --> ADD
      2. Private IP: Auto-Allocate
      3. IP Protocol: V4 Only
      4. VIP Address Allocation Network: PUBLIC - 0.0.0.0/0
      5. SAVE
   5. SAVE
6. Select Servers
   1. Select: "IP Address, Range, or DNS Name"
   2. Server IP Address: Enter the target VKS node IP (kubectl get vm -A -owide)
   3. Click "Add Server"
   4. SAVE

 

Check the SSH connection

Verify that the health status of the created Virtual Service is Green.

SSH to the target VIP. This sample tried to connect the VKS node in the VPC.

# Jumpserver
VSPHERE_NS=vsphere-ns-1    # vSphere Namespace
VKS_CLUSTER=test-1         # VKS Cluster name
AVI_VS_VIP=192.168.x.x     # Virtual Service Address

# Switch to the target vsphere namespace
kubectl config use-context <TARGET VSPHERE_NAMESPACE>

# Generate a SSH key
kubectl -n "${VSPHERE_NS}" get secret "${VKS_CLUSTER}-ssh" -o jsonpath='{.data.ssh-privatekey}' | base64 -d > vks_ssh_key
chmod 600 vks_ssh_key

# SSH to the target vks cluster node
ssh -i vks_ssh_key vmware-system-user@"${AVI_VS_VIP}"

Once no longer needed, delete the Virtual Service.

Additional Information

Japanese KB: https://knowledge.broadcom.com/external/article/436621