Checklist for verifying complete RAN (DU/CU) deployment Telco Cloud Automation (TCA) 2.3 / TCP RAN 3.0
Article ID: 325394
Updated On:
Products
VMware
VMware Telco Cloud Automation
VMware Telco Cloud Platform RAN
Issue/Introduction
This document is intended to serve as a checklist to be confirmed after a successful RAN Network Function deployment. VMware considers a Network Function (DU/CU) to be complete and in a supported state only if all the applicable conditions in this article are met.
Environment
VMware Telco Cloud Automation 2.3
Resolution
DU Server BIOS
CU VMs (both CU-CP and CU-UP) run regular (non-real-time) workloads on the Photon Linux (generic) kernel (non-real-time).
A non-real-time system such as a CU should NOT have real-time application threads. This can be verified using ps or top.
Example:
ps -eo cmd,pid,rtprio | grep -v -e "\[" -e "-$"
This prints the command-name, PID and Real-Time priority of all processes, and skips kernel threads (which are always in square brackets []) and those that do not have a real-time priority (indicated by a “-”, as opposed to a priority number. This list should come up empty, which indicates that there are no real-time application threads in the system.
KB article for updating Photon rt-kernel versions from what is built into TCA: KB 87717
Need to login to ESXi host as root. Reboot is required after modification.
| VMware Validated BIOS | |
| Dell XR11 (Icelake) | 1.6.5 |
| Supermicro (Icelake) | 3.4 v2.0 |
| Dell R740 (Cascadelake) | 2.11.2 |
| Dell R750 (Icelake) | 1.6.5 |
| BIOS Configuration for Dell Icelake Servers - BIOS 1.0.2 or higher (with Telco profile option). | ||
| XR11 BIOS -> System Profile Settings ->Workload Profile | Telco profile | Set all BIOS parameters to match telco RAN requirements |
Kubernetes cluster
| Telco Cloud Automation 2.1 | Kubernetes 1.22.17, 1.23.16, 1.24.10 | Tanzu Kubernetes Grid 2.1.1 |
Telco Cloud Automation
| Item | Expected Value | Notes |
| Cluster Status | Active | Cluster also should not display any warnings or errors. |
| Nodepool Status | Active | |
| Nodepool CPU Manager Scheduling policy | Static | Set CPU reservations on the Worker nodes as Static for CPU-intensive workloads. At least 1 CPU is reserved for Kubernetes Processes (eg. kubelet, kube-proxy), at least 1 CPU is reserved for System Processes (eg. sshd). If vendor has additional pods for non L1/L2 or non-real time applications. 2 CPUs can be reserved for Kubernetes Processes and 2 CPUs for System Processes. If 2 CPUs are reserved for Kubernetes Processes and System Processes, it is recommended to set isolcpus to be 2-(n-1). In CSAR, it can be configured as “2-{{tca.node.vmNumCPUs}}”. If 4 CPUs are reserved for Kubernetes Processes and System Processes, the isolcpus can be configured as “4-{{tca.node.vmNumCPUs}}”. At least 2 GB of Memory should be reserved, 1 GB for system processes and 1 GB for Kubernetes processes. |
| NF Catalog Status | Onboarding State: Onboarded Operational State: Enabled |
| NF CSAR Content | infra_requirements must be defined correctly. | Make sure the customizations are defined correctly under infra_requirements Example: topology_template: substitution_mappings: node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 node_templates: du-1.8: node_type: tosca.nodes.nfv.VMware.CNF.du-1.8 properties: descriptor_id: nfd_testing provider: xyz vnfm_info: - gvnfmdriver infra_requirements: node_components: |
| Isolcpus in the infra-requirements section | Should start with 2 e.g., 2-xx | Dependent on static CPU manager policy CPU configuration (for 1+1, should start from 2, in case of 2+2, start from 4) |
| Stalld package version | stalld-1.14.1-3 or higher | |
| Stalld configuration for Logging, Threshold and Boost Duration | In /etc/sysconfig/stalld: LOGGING= THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. Example: THRESH=”-t 2” (2 seconds) BD=”-d 1” (1 second) | LOGGING should NOT be set to: LOGGING=--log_only |
| Tuned package version | 2.15.0-3.ph3 or higher | |
| Tuned profile | Profile: realtime /etc/tuned/realtime-variables.conf should have the same set of isolated_cores as configured in the kernel command-line (/proc/cmdline) |
| Virtual Infrastructure | Connection Status: Connected | Connection status for that virtual infrastructure should be Connected before instantiating the CNF catalog |
| NF Instantiation Status | Instantiated | Once the Network Function is instantiated check the status: status: Instantiated |
ESXi Configuration
| Item | Expected Value |
| ESXi Power management policy | High performance |
Recommended Versions for ESXi, ESXi Drivers and Firmware and Photon OS Driver.
| Component | Vendor | Version |
| ESXi | VMware | VMware ESXi 8.0.0 21203435 |
| TCA | VMware | 2.3 |
| vCenter | VMware | vCenter Server 8.0.0b 21216066 |
| PhotonOS | ||
| PhotonOS | VMware | "3.0" |
| RT Kernel | VMware | 4.19.272-4 or higher |
| vmxnet3 | VMware | 1.7.0.0-k-NAPI |
| ice (PCI passthru) | Intel/VMware | 1.11.14 (for NVM4.2, with rt kernel 4.19.272-4) |
| iavf | Intel/VMware | 4.8.2 (for NVM4.2, with rt kernel 4.19.272-4) |
| i40e (PCI passthru) | Intel/VMware | 2.22.18 |
| Kubernetes | ||
| Tanzu Kubernetes Group | VMware | 2.1.1 |
| client | VMware | 1.22.17, 1.23.16, 1.24.10 |
| server | VMware | 1.22.17, 1.23.16, 1.24.10 |
| ESXi Driver (Intel) | ||
| XXV710 | Intel/VMware | i40en 2.1.5.0 |
| E810-C | Intel/VMware | Icen_1.11.3.0 |
| NIC Firmware | ||
| XXV710 | Intel | 8.40 0x8000b1fb 20.5.13 |
| E810 | Intel | NVM 4.20 0x80017789 1.3346.0 |
| ACC100 | Intel | |
| ibbd-pf | Intel | 1.0.8-1OEM.700.1.0.15843807_20100251 |
| ibbd-tools | Intel | 1.0.8-1OEM.700.1.0.15843807_18401717 for ESXi 7.0 1.0.8-1OEM.800.1.0.20613240_20922857 for ESXi 8.0 |
| DPDK | Intel | 20.11 , 21.11, 22.11 |
| stalld | 1.3.0-9 | |
| tuned | 2.15.0-3 | |
| linuxptp | 3.1.1-3 | |
| Silicom STS | Silicom | NIC FW: 4.01 0x80014757 1.3256.0 tsyncd 2.1.2.11 ESXi driver: icen_1.9.5 iavf: 4.5.3 ice: 1.9.11 |
Photon / VM configuration for CU:
CU VMs (both CU-CP and CU-UP) run regular (non-real-time) workloads on the Photon Linux (generic) kernel (non-real-time).
| Item | Command | Expected Value | Notes |
| Kernel Flavor | uname -a | 4.19.xxx-x.ph3 | The kernel version suffix ending with .ph3 indicates that the kernel flavor is Linux (generic). |
| Kernel Version (TCA 2.3) | uname -a | 4.19.272-5.ph3 | |
| Tuned Version | rpm -qa tuned | 2.15.0-3.ph3 or higher | The use and installation of tuned with the workload is up to the CU vendor. |
| Tuned Profile | tuned-adm active | cpu-partitioning or N/A | The tuned profile depends on what the CU vendor selected to use with the workload. |
A non-real-time system such as a CU should NOT have real-time application threads. This can be verified using ps or top.
Example:
ps -eo cmd,pid,rtprio | grep -v -e "\[" -e "-$"
This prints the command-name, PID and Real-Time priority of all processes, and skips kernel threads (which are always in square brackets []) and those that do not have a real-time priority (indicated by a “-”, as opposed to a priority number. This list should come up empty, which indicates that there are no real-time application threads in the system.
Photon / VM Configuration for DU:
DU VMs run a mix of real-time and non-real-time workloads on the Photon linux-rt kernel (real-time).| Item | Command | Expected Value | Notes |
| Kernel Flavor | uname -a | 4.19.xxx-rtxx-xx.ph3.rt | The kernel version suffix ending with .ph3-rt indicates that the kernel flavor is linux-rt (real-time). |
| Kernel Version (TCA 2.3) | uname -a | 4.19.272-4.ph3.rt | Update using KB87717, KB90345 if different |
| Kernel Command-Line Parameters | cat /proc/cmdline | The kernel command-line MUST have the following values: isolcpus=<list-of-isolated-CPUs> irqaffinity=<other-than-isolated-CPUs> rcu_nocbs=<list-of-isolated-CPUs> nohz=on nohz_full=<list-of-isolated-CPUs> nosoftlockup nowatchdog nmi_watchdog=0 idle=halt intel_idle.max_cstate=0 cpuidle.off=1 mce=off or mce=ignore_ce (Either one is valid) audit=0 cgroup.memory=nokmem The kernel command-line MAY have the following values to achieve lower latency: intel_pstate=disable skew_tick=1 tsc=reliable softlockup_panic=0 selinux=0 enforcing=0 | |
| Tuned Version | rpm -qa tuned | 2.15.0-3.ph3 or higher |
| Tuned Profile | Command 1: tuned-adm active Command 2: /proc/cmdine | Output for command 1: realtime or a derived profile specific to the Network Function vendor. Output for command 2: The output must include the command-line parameters configured in /boot/tuned.cfg (which is populated by the chosen tuned profile). This indicates that any bootloader parameters configured by the tuned profile are also active (post reboot). | The tuned profile could be ‘realtime’ or a custom profile derived from the real time profile shipped by the Network Function vendor. |
| Stalld Version | rpm -qa stalld | 1.14.1-0 or higher | |
| Stalld Configuration | cat /etc/sysconfig/stalld | The LOGGING= field should be blank, like shown below: LOGGING= THRESH should be at most 5 seconds and BD (Boost Duration) should be less than THRESH. Eg: THRESH=”-t 2” (2 seconds) and BD=”-d 1” (1 second) | LOGGING should NOT be set to: LOGGING=--log_only By default, the stalld package configuration is set to logging only (no priority boosting). This should be changed by the Network Function vendor to make it fully active and not just leave it in logging-only mode. Also, the starving threshold should be small for stalld to detect and revive the system from potential starvation. |
| Stalld Service Status | systemctl status stalld | The output must include: Active: active (running) | |
| Ptp systemd service status | Systemctl status ptp4l Systemctl status phc2sys | The output must include: Active: active (running) | |
| PTP drift status | pmc -u -d <domainNumber> -b 0 'GET TIME_STATUS_NP' | The output should show gmpresent:true Should also list gmIdentity Typical output would look like this: 40a6b7.fffe.0deb61-0 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP master_offset -6218 ingress_time 1614359358101173060 cumulativeScaledRateOffset +0.000000000 scaledLastGmPhaseChange 0 gmTimeBaseIndicator 0 lastGmPhaseChange 0x0000'0000000000000000.0000 gmPresent true gmIdentity fcaf6a.fffe.02e8f4 | Replace <domainNumber> properly. E.g., pmc -u -d 24 -b 0 'GET TIME_STATUS_NP' |
| PTP Clock status | pmc -u -b 0 -d <domainNumber> 'GET PORT_DATA_SET' | The output should show ‘portState’ as SLAVE. # pmc -u -b 0 -d 24 'GET PORT_DATA_SET' sending: GET PORT_DATA_SET 000c29.fffe.666be9-1 seq 0 RESPONSE MANAGEMENT PORT_DATA_SET portIdentity 000c29.fffe.666be9-1 portState SLAVE logMinDelayReqInterval -4 peerMeanPathDelay 0 logAnnounceInterval -3 announceReceiptTimeout 3 logSyncInterval -4 delayMechanism 1 logMinPdelayReqInterval 0 versionNumber 2 | Replace <domainNumber> properly. |
| vHT setting | Check vHT/SMT tag is added in csar file | “enableSMT: true” is added in csar file and CNF is instantiated |
KB article for updating Photon rt-kernel versions from what is built into TCA: KB 87717
Memory optimization configuration for VM:
ESXi commands to free up additional memory if using low memory servers (64GB RAM)Need to login to ESXi host as root. Reboot is required after modification.
| Item | Command | Expected Value | Notes |
| Intel Icen 1.8.5 | esxcli software vib install -d /<location of the downloaded driver>/Intel-icen_1.8.5.0-1OEM.xxx.zip | Icen driver is upgraded to 1.8.5 | NOTE: Intel icen 1.8.5 or higher reduces memory consumption. |
| Intel ACC100 Ibbd-pf | esxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-pf-1.0.8-1OEM.xxx.zip | ACC100 ibbd-pf driver is upgraded to 1.0.8 | |
| Ibbd-tool | esxcli software component apply –d /<location of the extracted driver>/VMW-esx-7.0.0-Intel-ibbd-tool-1.0.8-1OEM.xxx.zip | ACC100 ibbd-tool driver is upgraded to 1.0.8 | |
| MaxVMs | esxcli system settings kernel set -s maxVMs -v 6 | MaxVMs = 6 | This should free up ~1GB |
| Disable DRSS/RSS | esxcfg-module icen -s 'DRSS=0,0,0,0 RSS=0,0,0,0 | DRSS = 0,0,0,0 RSS = 0,0,0,0 | This should free up ~450 GB |
Feedback
Yes
No
Powered by
