How to upgrade Photon Kernel when running Telco Cloud Automation (TCA)

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Note: For details on enabling additional Photon-RT Kernel versions in TCA, see Enabling Additional Photon-RT Kernel Versions in Telco Cloud Automation.  

Photon Kernel versions in the 198 series, with 4.19.198-18 version and above as well as kernel versions in the 264 series with 4.19.264-6 and above come with sub-packages for different Intel ice/iavf driver versions.

These sub packages correspond to different Intel E810 NIC firmware (NVM) versions. This gives a user the flexibility to specify which sub-package to select based on the NVM version they have deployed.

Users need to update their CSAR with selected Intel ice/iavf driver versions as described in this document. 
 

NVM version	ice version	iavf version
3.0	linux-rt-drivers-intel-ice-1.6.4	linux-rt-drivers-intel-iavf-4.2.7
3.2	linux-rt-drivers-intel-ice-1.8.3	linux-rt-drivers-intel-iavf-4.4.2
4.0	linux-rt-drivers-intel-ice-1.9.11	linux-rt-drivers-intel-iavf-4.5.3
4.2	linux-rt-drivers-intel-ice-1.11.14	linux-rt-drivers-intel-iavf-4.8.2

The following scenarios are covered below:  

• Kernel version with no sub packages to a kernel version with sub packages (198-18 or higher).
• Kernel version with default sub packages to a newer kernel version with default sub packages (198-18 or higher)
• Kernel version with sub packages to newer kernel version with the same version of sub packages 
• Rollback procedure 

Environment

VMware Telco Cloud Automation 2.1.1
VMware Telco Cloud Automation 2.1
VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 2.0

Resolution

Case 1: Kernel version with no sub packages to a kernel version with sub packages (198-18 or higher)

	Current	New 198-21 with sub-packages
Kernel	4.19.198-5.ph3	4.19.198-21.ph3
Drivers	ice iavf	linux-rt-drivers-intel-ice-1.9.11  linux-rt-drivers-intel-iavf-4.5.3

The following steps can be taken to upgrade Photon kernels version 4.19.198-14 or below to 4.19.198-21 and higher with updated drivers. In this example kernel 4.19.198-5 is being updated to 4.19.198.21.

1. Follow the steps in KB article 87717 to add kernel version 4.19.198-21 to TCA if it is not already supported.
2. The following steps are to check what driver versions are currently included in the Photon package and to apply the corresponding driver patch.
   1. Go into a worker node where the Network Function has been deployed and list the current driver details with the commands modinfo ice and modinfo iavf.          
   2. These are the drivers in kernel 198-5 that will need to be updated: 

      Current Driver	New Driver to be installed with 4.19.198-21
      ice 1.6.4	linux-rt-drivers-intel-ice-1.9.11-4.19.198-21
      iavf 4.2.7	linux-rt-drivers-intel-iavf-4.5.3-4.19.198-21

3. Make the necessary changes to the CSAR to add the new drivers by logging into the TCA Manager UI.
   1. From the Network Function section under Catalog, select the Catalog Name.
   2. Click EDIT and go to Resources.
       
4. Edit Definitions/VNFD.yaml file.
   1. In the CSAR, change Kernel version from 4.19.198-5.ph3 to 4.19.198-21.ph3.
       
   2. Scroll down to custom_packages and add the new ice and iavf drivers as shown below:
   3. Save the changes made to the CSAR by clicking SAVE.
   4. Next, click SAVE AS NEW so that this edited CSAR is saved in the catalog with a different name than the one currently installed.
      
       
5. Terminate the current network function using the following steps.
   1. Go to the Inventory Network Function page and terminate the network function with kernel version 198-5. Note: Terminating the network function will stop the pod.
   2. Click Terminate. Wait for the termination to complete.
   3. Instantiate the network function with the new CSAR created in Step 3.
      Note: Instantiating a new NF with the updated CSAR will reboot the VM
       
6. On the worker node, Verify that the updated drivers have been installed by running the following command:
   tdnf list installed | egrep 'ice|iavf

Case 2: Kernel version with default sub packages to a newer kernel version with default sub packages (198-18 or higher)

	Current	New 198-21 with sub-packages
Kernel	4.19.198-18.ph3	4.19.198-21.ph3
Drivers	linux-rt-drivers-intel-ice- 1.8.3  linux-rt-drivers-intel-iavf- 4.4.2	linux-rt-drivers-intel-ice-1.9.11  linux-rt-drivers-intel-iavf-4.5.3

1. Verify installed drivers with the following command:
   tdnf list installed | grep linux-rt
   
    
2. Use the following command to remove 4.19.198-18 and its sub packages:
   tdnf remove linux-rt.x86_64 
3. When prompted Is this OK [y/N]?, enter y to proceed.
4. Terminate the network function in TCA by following step 4 under procedure 1. 
5. Create new CSAR and add the kernel version to the ice and iavf names. 
   
    

Case 3:  Kernel version with sub packages to newer kernel version with the same version of sub packages 

	Current with sub-packages	New 198-21 with sub-packages
Kernel	4.19.198-18.ph3	4.19.198-21.ph3
Drivers	linux-rt-drivers-intel-ice-1.6.4  linux-rt-drivers-intel- iavf-4.4.2	linux-rt-drivers-intel-ice-1.6.4  linux-rt-drivers-intel- iavf-4.4.2

 In this case there is no need to manually remove the sub-packages as they are the same sub package version by TCA.
 

Rollback Procedure

 The steps below are required only if there is a need to revert the drivers to a version that is 4.19.198-14 or below. 

Example: Downgrading an NF from linux-rt kernel version 4.19.198-21 to version 4.19.198-5:

1. Confirm the version 4.19.198-21 ice/iavf linux-rt kernels are installed on the worker node by running the following command:
   uname -a
    
2. Obtain the names of the linux-rt kernel sub-packages by running the following command:
   rpm -qa | grep linux-rt
   
   If installed, the results will present the expected kernel version as shown below:

   root [ ~ ]# rpm -qa | grep linux-rt  
   linux-rt-drivers-intel-ice-1.6.4-4.19.198-21.ph3.x86_64  
   linux-rt-drivers-intel-iavf-4.2.7-4.19.198-21.ph3.x86_64  
   linux-rt-4.19.198-21.ph3.x86_64
   

3. Remove the 4.19.198-21 linux-rt kernel sub-packages by running the following command:
   tdnf remove linuxRtName
   
   Note: Replace linuxRtName in the command above with the name of the linux-rt subpackage obtained in step 2.
    
4. Confirm the linux-rt kernel subpackage is no longer installed by running the following command:
   rpm -qa | grep linux-rt  
    
5. Run the following command to confirm only the main linux-rt kernel is installed: 
   rpm -qa | grep linux-rt  
    
6. Terminate the 4.19.198-21 network function in TCA by following step 4 under Procedure 1. 
7. Go to the TCA UI and instantiate the desired Network Function CSAR with the 198-5 kernel.  
8. Confirm the 4.19.198-5 linux-rt kernel version has replaced the 4.19.198-21 version by running the following command:
   uname -a

Additional Information

Note:  Not specifying ice and iavf in the CSAR will produce an error mentioning ptp4l.service: 

Failed. Error: vmconfig status is Normal, nodeconfig status is Failed.nodeconfig failed: [{Plugin dpdkBind is in Waiting stage, reason: waiting for status of plugin kernelMods become Normal, lastError: }{Plugin sriovdp is in Waiting stage, reason: waiting for dpdk binding finished, lastError: }{Plugin systemdsrv is in Failed stage, reason: Please check the plugin reason for details, lastError: start failed}] 

dpdkBind - Reason
waiting for status of plugin kernelMods become Normal 
sriovdp - Reason: 
waiting for dpdk binding finished 
systemdsrv - Reason: 
● ptp4l.service - Precision Time Protocol (PTP) service 
   Loaded: loaded (/lib/systemd/system/ptp4l.service; enabled; vendor preset: enabled) 
   Active: failed (Result: exit-code) since Thu 2023-03-30 21:48:56 UTC; 5s ago 
  Process: 3028 ExecStart=/usr/sbin/ptp4l $OPTIONS (code=exited, status=255) 
 Main PID: 3028 (code=exited, status=255)