Using nsenter for troubleshooting kubernetes pods
search cancel

Using nsenter for troubleshooting kubernetes pods

book

Article ID: 389166

calendar_today

Updated On: 05-05-2025

Products

Tanzu Kubernetes Grid

Issue/Introduction

The nsenter command executes program in the namespace(s) that are specified in the command-line options. If program is not given, then "${SHELL}" is run (default: /bin/sh).

  • IPC namespace
    IPC namespaces isolate certain IPC resources: System IPC identifiers and its own POSIX message queue file system. In one IPC namespace, objects that are created in this namespace are visible to all other processes that are members of this namespace, but are invisible to processes in other IPC namespaces.
  • UTS namespace
    UTS namespaces isolate two system identifiers: the hostname and the NIS domain name. In one UTS namespace, changes made to these identifiers are visible to all other processes in the same UTS namespace, but are invisible to processes in other UTS namespaces.
  • Network namespace
    Network namespaces isolate system resources that are associated with networking. Those resources include network devices, IPv4 and IPv6, protocol stacks, IP routing tables, firewall rules, and other networking-related resources. In one network namespace, processes in these namespaces have independent networking resources such as IPv4 and IPv6 stacks, IP routing tables, and firewall rules, as compared with the processes in other network namespaces.
  • PID namespace
    PID namespaces isolate the process ID number space, which means that processes in different PID namespaces can have the same PID. PID namespaces allow containers to provide functions such as suspending or resuming the set of processes in the container and migrating the container to a new host while the processes inside the container maintain the same PIDs.
  • Mount namespace
    Mount namespaces isolate the list of mount points that are seen by the processes in each namespace instance. The processes in each mount namespace instance will see distinct single-directory hierarchies.

Environment

Tanzu Kubernetes Grid 2.x.x

Cause

nsenter is a Linux utility that allows you to enter namespaces of other processes. In a Kubernetes cluster, each pod has its own network namespace, which means that you can use nsenter to enter the network namespace of a pod and troubleshoot networking issues from the host node of the pod.
This is extremely useful in scenarios where pods do not have a shell to exec into or in environments where you might not have access to a network utility pod to troubleshoot. 

Resolution

Issue 1 : Unable to identify which process(container) is consuming resource from the host node. 

- Use the top command to identity the process consuming high resources from within the host node.

- Use the ps -ef command to identify the PID of the container from within the host node.

root@workload:~#ps -ef | grep trivy
UID          PID          PPID       C STIME TTY          TIME           CMD
10000     639589  639059  0 Feb25 ?        00:04:17 /home/scanner/bin/scanner-trivy


- Use the lsns command to list the namespaces associated with a given process. 

root@workload:~# lsns -p 639589
 NS    TYPE   NPROCS    PID USER  COMMAND
4026531834 time      289      1 root  /sbin/init splash
4026531837 user      288      1 root  /sbin/init splash
4026533742 net         2 639079 10000 /pause
4026533810 uts         2 639079 10000 /pause
4026533811 ipc         2 639079 10000 /pause
4026533819 mnt         1 639589 10000 /home/scanner/bin/scanner-trivy
4026533820 pid         1 639589 10000 /home/scanner/bin/scanner-trivy
4026533821 cgroup      1 639589 10000 /home/scanner/bin/scanner-trivy


- Inspect the namespaces with nsenter. The nsenter command expands to namespace enter. It accepts different options to only enter the specified namespace. We can use the following command to enter the networking namespace and identify the pod using the ip a command. 

root@workload:~#nsenter -t 639589 -n ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@#####: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ##:##:##:##:##:## brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.96.3.152/24 brd 100.96.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 ####::####:####:####:####/64 scope link 
       valid_lft forever preferred_lft forever

here, -t is the target process id, and -n refers to the network namespace. 

- From the above output we got the IP Address associated to the pod, We can use this to identity the pod running on the cluster by running the following command from the Jump server. 

ubuntu@jumpbox:~$ kubectl get pods -A -o wide  | grep 100.96.3.152
tanzu-system-registry   harbor-trivy-0                                                        1/1     Running     0               24h     100.96.3.152     workload-node-1     <none>           <none>



- We can use the -p flag to connect to the process namespace to check the process details 

root@workload:~#nsenter -t 639589 -p -r ps -ef
UID              PID    PPID C STIME TTY          TIME CMD
scanner            1       0 0 04:32 ?        00:03:12 scanner-trivy
root             102       0 0 28:43 pts/1    00:00:00 ps -ef

- We can use the -u flag to connect to the UTC namespace to check the hostname. 

root@workload:~#nsenter -t 639589 -u hostname
harbor-trivy-0



Additional Information

Powered by Wolken Software