Wavefront collector service account cannot list ReplicationControllers & Secrets in a TKGI cluster
search cancel

Wavefront collector service account cannot list ReplicationControllers & Secrets in a TKGI cluster

book

Article ID: 298725

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

This issue was reported in an environment with the following versions:
  • Ops Manager - v2.10.16-build.269
  • TKGI - v1.11.3
  • Wavefront collector - wavefronthq/wavefront-kubernetes-collector:1.3.4
  • Wavefront proxy - wavefronthq/proxy:9.7
The wavefront collector pods show errors that the wavefront-collector service account cannot perform List operations on ReplicationControllers and Secrets. For example:
E0915 23:05:13.075556       1 reflector.go:125] github.com/wavefronthq/wavefront-collector-for-kubernetes/plugins/sources/kstate/lister.go:98: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:pks-system:wavefront-collector" cannot list resource "replicationcontrollers" in API group "" at the cluster scope

E0915 23:07:28.668873       1 reflector.go:125] github.com/wavefronthq/wavefront-collector-for-kubernetes/plugins/discovery/config.go:268: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:pks-system:wavefront-collector" cannot list resource "secrets" in API group "" in the namespace "pks-system"

This issue happens because the cluster role created for the wavefront-collector does not have the required permissions. 

Note - No issues were reported in relation to wavefront workload not being operational due to this issue leading to breaking the integration between TKGI & Wavefront. 


Environment

Product Version: 1.11
OS: Linux

Resolution

Note - If you are on v1.11.5 & greater or v1.12.x, this issue is already fixed in those releases. The following instructions are for working around this issue if you are not on a TKGI release where this issue has not been fixed.

Workaround
  • Edit the ClusterRole called wavefront-collector and make sure that the rule matches the following (note replicationcontrollers & secrets are added in the resources array):
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - nodes/stats
  - pods
  - replicationcontrollers
  - secrets
  - services
  verbs:
  - get
  - list
  - watch
  • Wait for the collector pods to reconcile and after a while, you will see that the errors regarding the service account not being able to list resources will not be reported anymore.