How to Modify Antrea’s OVS Installation logic on TKG Windows clusters

Products

• VMware

Issue/Introduction

Symptoms:

You see the following logs inside of the installation logs for C:/k/antrea/, which indicate that the windows nodes were never able to finish installing OVS.

Error: Found existing SSL library. \n\r\n8/25/2023 3:30 PM Found existing SSL library. \n\r\n8/25/2023 3:30 PM Download SSL files failed, URL: https://indy.fulgan.com/SSL/openssl-1.0.2u-x64_86-win64.zip

This was caused by failure in pulling the SSL Libraries from the internet.

Resolution

Configuring antrea to pull SSL libraries from either a place on the internet which is more trusted by your organization, or else, from a local repo of your own choosing. 

How to modify the Antrea Windows installation on TKG to use a custom SSL library in an Internet-Restricted environment:

*Prerequisites and Assumptions

• All activities below are performed on a Linux operating system which contains a suitably common set of binaries
• cURL is available on your custom Windows image

The SSL libraries which are required for TKG’s antrea OVS installation can be pulled down to an internet-restricted location when needed. Here’s how to set this up:
 

1. Download the Install-OVS.ps file to a webserver that is local (i.e. python SimpleHTTPServer example)

This KB article contains an attachment of a powershell file you can host locally and securely on any webserver.  

• NOTE: If you have no webserver to use and cannot access the above URL from your location, you can 

  • Create a directory on any standard linux machine

  • Run python -m SimpleHTTPServer 8000, do host the files in your local directory on that machines IP.

  • Reference the URL of your VM:8000/install-ovs-2.ps1, in the curl commands that occur later in this document.
     

2. Determine the default ClusterClass for your release, and make a new one

The goal of this article is specifically, to tell windows nodes, when bootstrapping, to overwrite the install-ovs powershell script with the custom one you have provided in step (1).

How to do this ? When making new TKG cluster installations, it is normal to customize bootstrapping of TKG by making custom cluster classes.  

A custom cluster class will give you the ability to change each and every runtime modification of your cluster, including the postKubeadmCommand invocations to install antrea on windows nodes.  
 

3. Create a YAML representation of your TKG Windows cluster

If you haven’t already, run 

kubectl get cluster my-existing-windows-cluster -o yaml > my-new-windows-cluster.yaml

We want to use the raw YAML representation of your cluster as our mechanism for making new TKG clusters, because we want to customize it at a fine granularity.
 

4. Create a copy of the ClusterClass, called TKG-Windows-ClusterClass or similar 

Copy your existing clusterclass to a new clusterclass, that you yourself can modify.  Existing TKG Clusterclasses that are managed by VMware do not allow customization.
 

kubectl get clusterclass -o yaml > my-new-windows-cluster-class.yaml

 

5. Modify the ClusterClass YAML file windows installation parts of it to “patch” the Install-OVS.ps script, by adding a curl command above the directive to Install-OVS.ps1

• Remove the Install-OVS.ps1 directive, and replace it with a new install-ovs-2.ps1 directive.   You likely will replace the URL in the example below with your own, internal URL which is accessible inside your firewall.

• As noted earlier, if you cannot access the ps1 file below, substitute the URL with a locally available, http downloadable copy of install-ovs-2.ps1. 

               # Install antrea-agent & OVS
               Import-Module C:\k\antrea\helper.psm1
               & Install-AntreaAgent -KubernetesHome "C:\k" -KubeConfig "C:\etc\kubernetes\kubelet.conf" -AntreaHome "C:\k\antrea" -AntreaVersion "1.11.1"
-              & C:\k\antrea\Install-OVS.ps1 -ImportCertificate $false -LocalFile C:\k\antrea\ovs-win64.zip

+              & curl.exe -kL https://raw.githubusercontent.com/jayunit100/k8sprototypes/master/tkg/install-ovs-2.ps1 -o C:\k\antrea\install-ovs-2.ps1
+              & C:\k\antrea\install-ovs-2.ps1 -ImportCertificate $false -LocalFile C:\k\antrea\ovs-win64.zip
               $nssm = (Get-Command nssm).Source
               & $nssm set kubelet start SERVICE_AUTO_START
               & $nssm install antrea-agent "C:\k\antrea\bin\antrea-agent.exe" "--config=C:\k\antrea\etc\antrea-agent.conf --logtostderr=false --log_dir=C:\var\log\antrea --alsologtostderr --log_file_max_size=100 --log_file_max_num=4"

 

6. Create the ClusterClass Kubernetes Object 

Now, run:

kubectl create -f my-new-windows-cluster-class.yaml

Confirm your cluster class has the new logic in it, by grepping out the “install-ovs-2-.ps1” string, i.e. You can run:  

kubectl get clusterclass tkg-vsphere-default-v1.1.1-windows -o yaml | grep install-ovs-2.ps1 

and you will see something such as: 

 

7. Modify the Cluster YAML file (my-new-windows-cluster.yaml) to point to the new clusterclass

In the below example, we change our cluster to point to a new cluster class called tkg-vsphere-default-v1.1.1-windows.

We are changing the topology.class field of the old cluster to my-new-windows-cluster.yaml deliberately, so that it uses our new powershell command, which downloads the a different OVS installation script, while TKG nodes are booting.
 

8. Create the Cluster object

Finally you can create your windows cluster in one step:

tanzu cluster create -f my-new-windows-cluster.yaml

 

9. Confirm your cluster came up.  Inspect logs as needed if your node doesn't fully bootstrap (ssh capv@<node-ip>). 

   Get-Content c:\k\antrea\install_ovs.log | Select-String ‘patches OVS’ | Out-Host -Paging
   

   Inside the node, you should see a string indicating it used a patched OVS Installation.

 

Additional Information

An informal walkthrough and video demonstration of the core concepts behind ClusterClass creation and modified OVS installation debugging is available here https://www.youtube.com/watch?v=61KSSBAXQFc&t=394s.