Managment vm has zombie processes related to anacron

Products

• VMware Integrated OpenStack

Issue/Introduction

• When running top on a management VM, there are zombie processes
  $ top -b -o +%CPU | head -n 35
top - 17:01:56 up 14 days, 23:23,  1 user,  load average: 8.80, 8.71, 8.40
Tasks: 1121 total,   7 running, 433 sleeping,   0 stopped, 681 zombie
...
  
  
• Check these zombie processes out to find which process is creating:
  # ps aux | awk '$8 ~ /^[Zz]/'
root       304  0.0  0.0      0     0 ?        Zs   Dec06   0:00 [anacron] <defunct>
root       316  0.0  0.0      0     0 ?        Zs   Dec06   0:00 [anacron] <defunct>
  ...

 

Environment

7.x

Cause

Inside the nova-vspc container, a cron job is used to rotate the vspc logs. A program called anacron periodically starts the logrotate job. It does so by running a child process in the background. Unfortunately, when the child process exits, the parent has already finished, thus leaving the child as orphaned.

Resolution

• We need to edit the statefulset for each compute:
  # kubectl get sts -n openstack  | grep compute
nova-compute-01-compute   1/1     2d6h
  
  
• For each compute statefulset returned we need to edit and add the line indicated below:
  #kubectl edit sts -n openstack <compute from above>

serviceAccount: nova-compute-01-compute
serviceAccountName: nova-compute-01-compute
shareProcessNamespace: true <<<< This makes all the containers in the pod to share the process namespace and needs to be added as the statefulset doesn't contain it.
  terminationGracePeriodSeconds: 30
  
  Note: Each statefulset (sts) will have a different service account name that matches the compute from the list above

The edit command will put you into vi.  Once you make the change above (shareProcessNamespace: true) and save/exit, the compute pod associated with that statefulset will terminate and restart.  You can watch that with the pods command.  This can be done with no real impact.