Unable to Provision Windows Operating System (OS) Worker Nodes in vSphere Supervisor

Products

VMware vSphere Kubernetes Service

Issue/Introduction

When trying to add a windows node-pool to a workload cluster in vSphere Supervisor cluster, one or more of the following errors are returned:

After saving edits to the workload cluster's YAML, similar errors to the following are returned:

Error from server (Forbidden): error when creating "<workload cluster name>": admission webhook "tkr-resolver-cluster-webhook.tanzu.vmware.com" denied the request: Could not resolve KR/OSImage 

Missing compatible KR/OSImage for the cluster

The worker nodes under the windows node-pool are stuck in Provisioning or Provisioned state.

Environment

vSphere Supervisor

vKR v1.31 and higher

Windows Server 2022

Cause

This issue can occur due to any of the below causes:

Misconfiguration of the node-pool annotations in the cluster YAML
- Although photon and ubuntu use os-name for the annotation, windows uses os-type
Cluster configuration requirements not met for Windows nodes
- Only worker nodes can use windows OS. Control plane nodes only support photon or ubuntu OS.
- The only Container Network Interface (CNI) that is supported for Windows nodes is Antrea.
The custom windows image or its content library is not recognized in the Supervisor cluster environment.

Resolution

Cluster YAML Steps

The annotation for specifying that a node-pool should use the Windows OS should match the below using os-type=windows:

- class: node-pool
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: os-type=windows

If your custom Windows image cannot be found by the system, you will need to specify the ID of the content library in the annotation similarly to the below:

- class: node-pool
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: os-type=windows, content-library=cl-<ID>

You can locate the content library ID by using the below commands:

Connect to the Supervisor cluster context
Retrieve a list of all content library items visible to the Supervisor cluster with their content library ID (cl-<ID>):
```
kubectl get contentlibraryitem -A

clitem-<ID> <image-name> cl-<ID> OVF
```
You can also list all content libraries and their IDs visible to the Supervisor cluster with the below command:
```
kubectl get contentlibrary -A

<namespace> cl-<ID> <name in vsphere> <Local/Subscribed>
```

After correcting the annotations, the next system creation of a worker node in the windows node-pool should succeed.

If it does not succeed, confirm that all requirements noted in the below documentation are met:

Provisioning TKG Clusters with Windows Node Pools

OSImage Steps

While connected to the Supervisor cluster context, the custom Windows image is not present as a osImage:

kubectl get osimages

Check that the Supervisor cluster is aware of the content library that has the custom Windows image:
```
kubectl get contentlibraryitem -A
kubectl get contentlibrary -A
```
In the vSphere web UI, navigate to Workload/Supervisor Management then to the namespace where you are deploying the windows nodepool.
Confirm that the content library containing the custom Windows image is attached to this namespace under the VM Service Card - Manage Content Libraries:
The system should automatically try to pull in and create osimages for content libraries attached to namespaces within the Supervisor environment.

Additional Information

Provisioning TKG Clusters with Windows Node Pools

Resolve VKR Image Conflicts

--------

"VMware will support issues with the vSphere Kubernetes Service Image Builder, but you are responsible for any issues relating to your image customizations and custom applications."

https://github.com/vmware/vks-image-builder

--------

Japanese version: vSphere Supervisor において Windows オペレーティングシステム（OS）のワーカーノードをプロビジョニングできない