Checklist for verifying complete RAN (DU/CU) deployment Telco Cloud Automation (TCA) 2.2 / TCP RAN 2.2
search cancel

Checklist for verifying complete RAN (DU/CU) deployment Telco Cloud Automation (TCA) 2.2 / TCP RAN 2.2

book

Article ID: 325393

calendar_today

Updated On:

Products

VMware VMware Telco Cloud Automation VMware Telco Cloud Platform RAN

Issue/Introduction

This document is intended to serve as a checklist to be confirmed after a successful RAN Network Function deployment. VMware considers a Network Function (DU/CU) to be complete and in a supported state only if all the applicable conditions in this article are met. 

Environment

VMware Telco Cloud Automation 2.2

Resolution

DU Server BIOS 
VMware Validated BIOS 
Dell R740 (Cascadelake) 2.11.2 
Dell XR11 (Icelake)  1.0.2 
Supermicro (Icelake)  3.4 v2.0 
  
BIOS Configuration for Dell Icelake Servers - BIOS 1.0.2 or higher (with Telco profile option). 
XR11 BIOS -> System
Profile Settings ->Workload Profile 
Telco profile Set all BIOS parameters to match telco RAN requirements 

Kubernetes cluster  
Telco Cloud Automation 2.1Kubernetes 1.20.15, 1.21.11, 1.22.9Tanzu Kubernetes Grid 1.5.4

Telco Cloud Automation
Item Expected Value Notes 
Cluster Status Active Cluster also should not display any warnings or errors. 
Nodepool Status Active  
Nodepool CPU Manager Scheduling policy Static Set CPU reservations on the Worker nodes as Static for CPU-intensive workloads.

At least 1 CPU is reserved for Kubernetes Processes (eg. kubelet, kube-proxy), at least 1 CPU is reserved for System Processes (eg. sshd).

If vendor has additional pods for non L1/L2 or non-real time applications. 2 CPUs can be reserved for Kubernetes Processes and 2 CPUs for System Processes. 

If 2 CPUs are reserved for Kubernetes Processes and System Processes, it is recommended to set isolcpus to be 2-(n-1). In CSAR, it can be configured as “2-{{tca.node.vmNumCPUs}}”. If 4 CPUs are reserved for Kubernetes Processes and System Processes, the isolcpus can be configured as “4-{{tca.node.vmNumCPUs}}”. 

At least 2 GB of Memory should be reserved, 1 GB for system processes and 1 GB for Kubernetes processes. 
NF Catalog Status Onboarding State: Onboarded 
Operational State: Enabled 
 
NF CSAR Content infra_requirements must be defined correctly. Make sure the customizations are defined correctly under infra_requirements 
Example: 
topology_template: 
  substitution_mappings: 
    node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 
  node_templates: 
    du-1.8: 
      node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 
      properties: 
        descriptor_id: nfd_testing 
        provider: xyz 
        vnfm_info: 
          - gvnfmdriver 
        infra_requirements: 
          node_components: 
Isolcpus in the infra-requirements section Should start with 2 
e.g., 2-xx 
Dependent on static CPU manager policy CPU configuration (for 1+1, should start from 2, in case of 2+2, start from 4) 
Stalld package version stalld-1.3.0-9 or higher  
Stalld configuration for Logging, Threshold and Boost Duration In /etc/sysconfig/stalld: 
LOGGING= 
 
THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. 
 
Example:
THRESH=”-t 2” (2 seconds)
BD=”-d 1” (1 second) 
LOGGING should NOT be set to: 
LOGGING=--log_only 
Tuned package version 2.15.0-3.ph3 or higher  
Tuned profile Profile: realtime 
  
/etc/tuned/realtime-variables.conf should have the same set of isolated_cores as configured in the kernel command-line (/proc/cmdline) 
 
Virtual Infrastructure Connection Status: Connected Connection status for that virtual infrastructure should be Connected before instantiating the CNF catalog 
NF Instantiation Status Instantiated Once the Network Function is instantiated check the status:
status: Instantiated  
 
ESXi Configuration 
Item Expected Value 
ESXi Power management policy High performance 
 
Recommended Versions for ESXi, ESXi Drivers and Firmware and Photon OS Driver. 
Component Vendor Version
ESXi VMware ESXi 7.0 Update 3 (7.0u3f) | Build  20036589
TCA VMware 2.1
vCenter VMware vCenter Server (7.0U3f) |Build 20051473
PhotonOS   
PhotonOS VMware "3.0" 
RT Kernel VMware 4.19.198-rt85-13.ph3.rt
vmxnet3 VMware 1.5.0.0-k-NAPI
ice (PCI passthru) Intel/VMware 1.8.3 
iavf Intel/VMware 4.4.2 
i40e (PCI passthru) Intel/VMware 2.15.9 
Kubernetes    
Tanzu Kubernetes GroupVMware1.5.4
client VMware 1.20.15, 1.21.11, 1.22.9
server VMware 1.20.15, 1.21.11, 1.22.9
ESXi Driver     
XXV710 Intel/VMware i40en 1.14.1.0
E810-C Intel/VMware icen_1.8.5.0
NIC Firmware     
XXV710 Intel 8.40 0x8000b1fb 20.5.13
E810 Intel NVM 3.2 (3.2 0x8000d833 1.3146.0)
ACC100 Intel  
ibbd-pf Intel 1.0.8-1OEM.700.1.0.15843807
ibbd-tools Intel 1.0.8-1OEM.700.1.0.15843807
DPDK Intel 20.11 
stalld  1.3.0-9
tuned 2.15.0-3
 
Photon / VM configuration for CU: 
CU VMs (both CU-CP and CU-UP) run regular (non-real-time) workloads on the Photon Linux (generic) kernel (non-real-time).  
Item Command Expected Value Notes 
Kernel Flavor uname -a 4.19.xxx-x.ph3 The kernel version suffix ending with .ph3 indicates that the kernel flavor is Linux (generic). 
Kernel Version (TCA 2.1uname -a CU-CP : 4.19.232-2 
CU-UP : 4.19.198-13.ph3
 
Tuned Version rpm -qa tuned 2.15.0-3.ph3 or higher The use and installation of tuned with the workload is up to the CU vendor. 
Tuned Profile tuned-adm active cpu-partitioning 
 or  
 N/A 
The tuned profile depends on what the CU vendor selected to use with the workload. 
 
A non-real-time system such as a CU should NOT have real-time application threads. This can be verified using ps or top.
Example: 
ps -eo cmd,pid,rtprio | grep -v -e "\[" -e "-$" 

This prints the command-name, PID and Real-Time priority of all processes, and skips kernel threads (which are always in square brackets []) and those that do not have a real-time priority (indicated by a “-”, as opposed to a priority number. This list should come up empty, which indicates that there are no real-time application threads in the system.

Photon / VM configuration for DU: 
DU VMs run a mix of real-time and non-real-time workloads on the Photon linux-rt kernel (real-time).  
Item Command Expected Value Notes 
Kernel Flavor uname -a 4.19.xxx-rtxx-xx.ph3.rtThe kernel version suffix ending with .ph3-rt indicates that the kernel flavor is linux-rt (real-time). 
Kernel Version
(TCA 2.1) 
uname -a 4.19.198-rt85-13.ph3.rtOther RT kernel versions are not recommended 
Kernel
Command-Line Parameters 
cat /proc/cmdline The kernel command-line MUST have the following values: 
isolcpus=<list-of-isolated-CPUs> 
irqaffinity=<other-than-isolated-CPUs> 
rcu_nocbs=<list-of-isolated-CPUs> 
nohz=on 
nohz_full=<list-of-isolated-CPUs> 
nosoftlockup  
nowatchdog 
nmi_watchdog=0  
idle=halt
intel_idle.max_cstate=0  
cpuidle.off=1  
mce=off or  mce=ignore_ce  (Either one is valid) 
 audit=0 
 cgroup.memory=nokmem 

The kernel command-line MAY have the following values to achieve lower latency: 
intel_pstate=disable  
skew_tick=1 
tsc=reliable 
softlockup_panic=0  
selinux=0 
enforcing=0 
 
Tuned Version rpm -qa tuned 2.15.0-3.ph3 or higher  
Tuned Profile Command 1: 
tuned-adm active 
 
Command 2: /proc/cmdine 
Output for command 1:  
realtime 
or 
a derived profile specific to the Network Function vendor. 
 
Output for command 2: 
The output must include the command-line parameters configured in /boot/tuned.cfg (which is populated by the chosen tuned profile).

This indicates that any bootloader parameters configured by the tuned profile are also active (post reboot). 
The tuned profile could be ‘realtime’ or a custom profile derived from the real time profile shipped by the Network Function vendor. 
Stalld Version rpm -qa stalld 1.3.0-9 or higher  
Stalld Configuration cat /etc/sysconfig/stalld The LOGGING= field should be blank, like shown below: 
 LOGGING= 
 
THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. 
 
Eg: THRESH=”-t 2” (2 seconds) and BD=”-d 1” (1 second) 
LOGGING should NOT be set to: 
LOGGING=--log_only 

By default, the stalld package configuration is set to logging only (no priority boosting).

This should be changed by the Network Function vendor to make it fully active and not just leave it in logging-only mode. 

Also, the starving threshold should be small for stalld to detect and revive the system from potential starvation. 
Stalld Service Status systemctl status stalld The output must include: Active: active (running)  
Ptp systemd service status Systemctl status ptp4l 
Systemctl status phc2sys 
The output must include: Active: active (running)  
PTP drift status pmc -u -d <domainNumber>  -b 0 'GET TIME_STATUS_NP' The output should show gmpresent:true 
Should also list gmIdentity 

Typical output would look like this: 
40a6b7.fffe.0deb61-0 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP 
master_offset -6218 
ingress_time 1614359358101173060 
cumulativeScaledRateOffset +0.000000000 
scaledLastGmPhaseChange 0 
gmTimeBaseIndicator 0 
lastGmPhaseChange 0x0000'0000000000000000.0000 
gmPresent true 
gmIdentity fcaf6a.fffe.02e8f4 
Replace <domainNumber> properly.
E.g., pmc -u -d 24 -b 0 'GET TIME_STATUS_NP' 
 
Memory optimization configuration for VM:
ESXi commands to free up additional memory if using low memory servers (64GB RAM)  
Need to login to ESXi host as root. Reboot is required after modification.
Item Command Expected Value Notes 
Intel Icen 1.8.5esxcli software vib install -d /<location of the downloaded driver>/Intel-icen_1.8.5.0-1OEM.xxx.zip
 
Icen driver is upgraded to 1.8.5 
Intel ACC100
Ibbd-pf
esxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-pf-1.0.8-1OEM.xxx.zip
 
ACC100 ibbd-pf driver is upgraded to 1.0.8 
Ibbd-toolesxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-tool-1.0.8-1OEM.xxx.zip
 
ACC100 ibbd-tool driver is upgraded to 1.0.8
 
 
MaxVMs esxcli system settings kernel set -s maxVMs -v 6MaxVMs = 6This should free up ~1GB
Disable DRSS/RSSesxcfg-module icen -s 'DRSS=0,0,0,0 RSS=0,0,0,0DRSS = 0,0,0,0
RSS = 0,0,0,0
This should free up ~450 GB