Gathering Logs via vks-support-bundler when requirePasswordOnSudo is Enabled
search cancel

Gathering Logs via vks-support-bundler when requirePasswordOnSudo is Enabled

book

Article ID: 442840

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Starting with VKS 3.5.0, you can customize 'requirePasswordOnSudo' in cluster configurations.
If you enable this feature, it prevents direct log collection via the 'vks-support-bundler' tool.

Environment

VMware vSphere Kubernetes Service >= 3.5.0

Cause

The 'vks-support-bundler' requires elevated privileges without a password prompt to execute log collection scripts on cluster nodes. When 'requirePasswordOnSudo' is active, the tool cannot complete these tasks.

Resolution

You can use the following workaround to gather logs by temporarily enabling `NOPASSWD` sudo.

1. Temporarily Enable NOPASSWD Sudo

You must create a DaemonSet to modify the sudoers files on cluster nodes.
**Note:** This DaemonSet changes the configuration on all nodes in the workload cluster. If you only enabled `requirePasswordOnSudo` on specific nodes, identify them in advance before applying the DaemonSet.

If your current image is not pullable, update it with a customer-pullable 'photon:5.0' image.

enable-nopasswd-sudo.yaml -

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: modify-sudoers
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: modify-sudoers
  template:
    metadata:
      labels:
        app: modify-sudoers
    spec:
      restartPolicy: Always
      volumes:
        - name: hostfs
          hostPath:
            path: /
      initContainers:
        - name: modify-sudoers
          image: photon:5.0
          command:
            - /bin/sh
            - -xc
            - |
              echo "Checking current sudoers configuration..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Modifying sudoers file to allow NOPASSWD..."
              chroot /host sed -i 's/^\([^ ]*\) ALL=(ALL) ALL/\1 ALL=(ALL) NOPASSWD:ALL/' /etc/sudoers.d/90-cloud-init-users
              echo "Verifying changes..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Done"
          securityContext:
            privileged: true
            runAsGroup: 0
            allowPrivilegeEscalation: true
            capabilities:
              add: ["ALL"]
          volumeMounts:
            - name: hostfs
              mountPath: /host
      containers:
        - name: sleep
          image: photon:5.0
          command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node.alpha.kubernetes.io/notReady
          operator: Exists
          effect: NoExecute
        - key: node.alpha.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
  • Save this YAML as 'enable-nopasswd-sudo.yaml'.
  • Apply it to the workload cluster: 'kubectl apply -f enable-nopasswd-sudo.yaml'.
  • Wait for the DaemonSet to complete on all nodes.

2. Collect Logs Using vks-support-bundler

Please refer to https://knowledge.broadcom.com/external/article/345464/gathering-logs-for-vsphere-with-tanzu.html for more info.

3. Revert Sudoers Configuration

3.1 If You Enabled requirePasswordOnSudo on All Nodes
Apply the following DaemonSet to revert the configuration:

revert-nopasswd-sudo.yaml -

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: revert-sudoers
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: revert-sudoers
  template:
    metadata:
      labels:
        app: revert-sudoers
    spec:
      restartPolicy: Always
      volumes:
        - name: hostfs
          hostPath: /
      initContainers:
        - name: revert-sudoers
          image: photon:5.0
          command:
            - /bin/sh
            - -xc
            - |
              echo "Checking current sudoers configuration..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Reverting sudoers file to require password..."
              chroot /host sed -i 's/^\([^ ]*\) ALL=(ALL) NOPASSWD:ALL/\1 ALL=(ALL) ALL/' /etc/sudoers.d/90-cloud-init-users
              echo "Verifying changes..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Done"
          securityContext:
            privileged: true
            runAsGroup: 0
            allowPrivilegeEscalation: true
            capabilities:
              add: ["ALL"]
          volumeMounts:
            - name: hostfs
              mountPath: /host
      containers:
        - name: sleep
          image: photon:5.0
          command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node.alpha.kubernetes.io/notReady
          operator: Exists
          effect: NoExecute
        - key: node.alpha.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute


3.2 If NOT all nodes are enabled with requirePasswordOnSudo
Please edit nodeAffinity to select nodes who need applying daemonSet. The example below shows the daemonSet will work on node with label 'vks.vmware.com/nodepool=worker' which will be reverted back to require password on sudo.

revert-nopasswd-sudo.yaml - 

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: revert-sudoers
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: revert-sudoers
  template:
    metadata:
      labels:
        app: revert-sudoers
    spec:
      restartPolicy: Always
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: vks.vmware.com/nodepool
                    operator: In
                    values:
                      - worker
      volumes:
        - name: hostfs
          hostPath:
            path: /
      initContainers:
        - name: revert-sudoers
          image: photon:5.0
          command:
            - /bin/sh
            - -xc
            - |
              echo "Checking current sudoers configuration..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Reverting sudoers file to require password..."
              chroot /host sed -i 's/^\([^ ]*\) ALL=(ALL) NOPASSWD:ALL/\1 ALL=(ALL) ALL/' /etc/sudoers.d/90-cloud-init-users
              echo "Verifying changes..."
              chroot /host cat /etc/sudoers.d/90-cloud-init-users 2>/dev/null
              echo "Done"
          securityContext:
            privileged: true
            runAsGroup: 0
            allowPrivilegeEscalation: true
            capabilities:
              add: ["ALL"]
          volumeMounts:
            - name: hostfs
              mountPath: /host
      containers:
        - name: sleep
          image: photon:5.0
          command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node.alpha.kubernetes.io/notReady
          operator: Exists
          effect: NoExecute
        - key: node.alpha.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
  • Modify the `revert-nopasswd-sudo.yaml` to include your specific node affinity.
  • Save and apply it:  'kubectl apply -f revert-nopasswd-sudo.yaml'.
  • Wait for the DaemonSet to complete.

4. Cleanup

You can now delete the temporary DaemonSets once the process is complete:

kubectl delete daemonset -n kube-system modify-sudoers
kubectl delete daemonset -n kube-system revert-sudoers

 

Additional Information

1. Gathering logs for vSphere with Tanzu
2. Knowledge Article 422111

 

Powered by Wolken Software