How to Modify Antrea’s OVS Installation logic on TKG Windows clusters

Products

VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

You see the following logs inside of the installation logs for C:/k/antrea/, which indicate that the windows nodes were never able to finish installing OVS.

Error: Found existing SSL library. \n\r\n8/25/2023 3:30 PM Found existing SSL library. \n\r\n8/25/2023 3:30 PM Download SSL files failed, URL: https://indy.fulgan.com/SSL/openssl-1.0.2u-x64_86-win64.zip

This was caused by failure in pulling the SSL Libraries from the internet.

Resolution

Configuring antrea to pull SSL libraries from either a place on the internet which is more trusted by your organization, or else, from a local repo of your own choosing.

How to modify the Antrea Windows installation on TKG to use a custom SSL library in an Internet-Restricted environment:

*Prerequisites and Assumptions

All activities below are performed on a Linux operating system which contains a suitably common set of binaries
cURL is available on your custom Windows image

The SSL libraries which are required for TKG’s antrea OVS installation can be pulled down to an internet-restricted location when needed. Here’s how to set this up:

Download the Install-OVS.ps file to a webserver that is local (i.e. python SimpleHTTPServer example)

This KB article contains an attachment of a powershell file you can host locally and securely on any webserver.

NOTE: If you have no webserver to use and cannot access the above URL from your location, you can
- Create a directory on any standard linux machine
- Run python -m SimpleHTTPServer 8000, do host the files in your local directory on that machines IP.
- Reference the URL of your VM:8000/install-ovs-2.ps1, in the curl commands that occur later in this document.

Determine the default ClusterClass for your release, and make a new one

The goal of this article is specifically, to tell windows nodes, when bootstrapping, to overwrite the install-ovs powershell script with the custom one you have provided in step (1).

How to do this ? When making new TKG cluster installations, it is normal to customize bootstrapping of TKG by making custom cluster classes.

A custom cluster class will give you the ability to change each and every runtime modification of your cluster, including the postKubeadmCommand invocations to install antrea on windows nodes.

Create a YAML representation of your TKG Windows cluster

If you haven’t already, run

kubectl get cluster my-existing-windows-cluster -o yaml > my-new-windows-cluster.yaml

We want to use the raw YAML representation of your cluster as our mechanism for making new TKG clusters, because we want to customize it at a fine granularity.

Create a copy of the ClusterClass, called TKG-Windows-ClusterClass or similar

Copy your existing clusterclass to a new clusterclass, that you yourself can modify. Existing TKG Clusterclasses that are managed by VMware do not allow customization.

kubectl get clusterclass -o yaml > my-new-windows-cluster-class.yaml

Modify the ClusterClass YAML file windows installation parts of it to “patch” the Install-OVS.ps script, by adding a curl command above the directive to Install-OVS.ps1

Remove the Install-OVS.ps1 directive, and replace it with a new install-ovs-2.ps1 directive. You likely will replace the URL in the example below with your own, internal URL which is accessible inside your firewall.
As noted earlier, if you cannot access the ps1 file below, substitute the URL with a locally available, http downloadable copy of install-ovs-2.ps1.

               # Install antrea-agent & OVS
               Import-Module C:\k\antrea\helper.psm1
               & Install-AntreaAgent -KubernetesHome "C:\k" -KubeConfig "C:\etc\kubernetes\kubelet.conf" -AntreaHome "C:\k\antrea" -AntreaVersion "1.11.1"
-              & C:\k\antrea\Install-OVS.ps1 -ImportCertificate $false -LocalFile C:\k\antrea\ovs-win64.zip

+              & curl.exe -kL https://raw.githubusercontent.com/jayunit100/k8sprototypes/master/tkg/install-ovs-2.ps1 -o C:\k\antrea\install-ovs-2.ps1
+              & C:\k\antrea\install-ovs-2.ps1 -ImportCertificate $false -LocalFile C:\k\antrea\ovs-win64.zip
               $nssm = (Get-Command nssm).Source
               & $nssm set kubelet start SERVICE_AUTO_START
               & $nssm install antrea-agent "C:\k\antrea\bin\antrea-agent.exe" "--config=C:\k\antrea\etc\antrea-agent.conf --logtostderr=false --log_dir=C:\var\log\antrea --alsologtostderr --log_file_max_size=100 --log_file_max_num=4"

Create the ClusterClass Kubernetes Object

Now, run:

kubectl create -f my-new-windows-cluster-class.yaml

Confirm your cluster class has the new logic in it, by grepping out the “install-ovs-2-.ps1” string, i.e. You can run:

kubectl get clusterclass tkg-vsphere-default-v1.1.1-windows -o yaml | grep install-ovs-2.ps1

and you will see something such as:

Modify the Cluster YAML file (my-new-windows-cluster.yaml) to point to the new clusterclass

In the below example, we change our cluster to point to a new cluster class called tkg-vsphere-default-v1.1.1-windows.

We are changing the topology.class field of the old cluster to my-new-windows-cluster.yaml deliberately, so that it uses our new powershell command, which downloads the a different OVS installation script, while TKG nodes are booting.

Create the Cluster object

Finally you can create your windows cluster in one step:

tanzu cluster create -f my-new-windows-cluster.yaml

Confirm your cluster came up. Inspect logs as needed if your node doesn't fully bootstrap (ssh capv@<node-ip>).
```
Get-Content c:\k\antrea\install_ovs.log | Select-String ‘patches OVS’ | Out-Host -Paging
```
Inside the node, you should see a string indicating it used a patched OVS Installation.

Additional Information

An informal walkthrough and video demonstration of the core concepts behind ClusterClass creation and modified OVS installation debugging is available here https://www.youtube.com/watch?v=61KSSBAXQFc&t=394s.

Attachments

Install-OVS get_app