k8s-metrics command is not working after upgrading to Aria Automation 8.18.1 Patch 5 or Orchestrator 8.18.1 Update 6

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Running the k8s-metrics command on any VMware Aria Automation or VMware Aria Orchestrator appliance node produces no output and returns a jq error. The metrics-server pods in the kube-system namespace remain in a Running or Ready state, indicating this is not a service outage.

jq: error (at <stdin>:1): null (null) has no keys
jq: error (at <stdin>:1): null (null) has no keys

The metrics-server pods (kube-system) remain Running/Ready - this is not an outage of metrics-server itself

Environment

• VMware Aria Automation 8.18.1 Patch 5
• VMware Aria Orchestrator 8.18.1 Update 6

Cause

The metrics-server component was updated from version v0.4.2 to v0.8.1. This new version requires the nodes/metrics RBAC permission to scrape metrics from kubelets, whereas previous versions used nodes/stats.

Resolution

A permanent fix is targeted for a future patch release. To receive updates on this issue, subscribe to this article.

Workaround:
This patch changes the "nodes/stats" RBAC grant to "nodes/metrics", both live in the cluster (for immediate effect) and in the on-disk chart template on every node (so the fix survives a future service restart, node reboot or chart re-deploy). The metrics-server picks up the live fix on its next scrape cycle (~10 seconds), no service restart needed

=============================================
Install patch
=============================================

1. Take simultaneous VM snapshots without memory of all vRA/vRO nodes if desired (this is a low-risk, additive RBAC change - no services are restarted, no downtime).
2. Run the following command once, on any single vRA/vRO node. It patches the on-disk chart template on every cluster node, applies the live RBAC fix once, and verifies that k8s-metrics reports data:

   base64 -d <<< IyEvYmluL2Jhc2gKIwojIFdvcmthcm91bmQgZm9yOiBrOHMtbWV0cmljcyByZXBvcnRzIG5vIGRhdGEgLyBqcSAibnVsbCAobnVsbCkgaGFzIG5vIGtleXMiCiMgYWZ0ZXIgdXBncmFkaW5nIHRvIEFyaWEgQXV0b21hdGlvbiA4LjE4LjEgUGF0Y2ggNSAvIE9yY2hlc3RyYXRvciBVcGRhdGUgNi4KIwojIFJvb3QgY2F1c2U6CiMgICBtZXRyaWNzLXNlcnZlciB3YXMgYnVtcGVkIHRvIHYwLjguMSBpbiB0aGlzIHBhdGNoLCB3aGljaCByZXF1aXJlcyB0aGUKIyAgICJub2Rlcy9tZXRyaWNzIiBSQkFDIHJlc291cmNlIGluc3RlYWQgb2YgIm5vZGVzL3N0YXRzIiB0byBzY3JhcGUKIyAgIGt1YmVsZXRzLgojCiMgVGhpcyBidWcgaXMgc3BlY2lmaWMgdG8gdGhpcyBvbmUgcGF0Y2ggcmVsZWFzZSAoOC4xOC4xIFA1IC8gdlJPIFU2KS4gVGhlCiMgcGVybWFuZW50IGZpeCBpcyBpbmNsdWRlZCBpbiB0aGUgbmV4dCBwYXRjaC4KIwojIEZpeDoKIyAgIC0gUGF0Y2ggdGhlIG9uLWRpc2sgY2hhcnQgdGVtcGxhdGUgYXQgL29wdC9jaGFydHMvcHJlbHVkZS1rOHMgb24gZXZlcnkKIyAgICAgY2x1c3RlciBub2RlICh2aWEgdnJhY2xpIGNsdXN0ZXIgZXhlYywgYmVsb3cpIHNvIHRoZSBmaXggc3Vydml2ZXMgdGhlCiMgICAgIG5leHQgcmVkZXBsb3kpLgojICAgLSBBcHBseSB0aGUgc2FtZSBjaGFuZ2UgbGl2ZSBmb3IgaW1tZWRpYXRlIGVmZmVjdCAoY2x1c3Rlci1zY29wZWQgUkJBQwojICAgICBvYmplY3QsIHBpY2tlZCB1cCBvbiBtZXRyaWNzLXNlcnZlcidzIG5leHQgc2NyYXBlLCB+MTBzLCBubyByZXN0YXJ0KS4KIyAgICAgVGhpcyBvbmx5IG5lZWRzIHRvIHJ1biBvbmNlLCBzbyBpdCBydW5zIGRpcmVjdGx5IGJlbG93IHJhdGhlciB0aGFuCiMgICAgIHRocm91Z2ggdnJhY2xpIGNsdXN0ZXIgZXhlYy4KIwojIFRoaXMgc2NyaXB0IGl0c2VsZiBvbmx5IG5lZWRzIHRvIHJ1biBvbmNlLCBvbiBhbnkgc2luZ2xlIGNsdXN0ZXIgbm9kZS4KIwoKc2V0IC11byBwaXBlZmFpbAoKbG9nRGlyPSIvdmFyL2xvZy92bXdhcmUvcHJlbHVkZSIKdGltZXN0YW1wPSQoZGF0ZSAnKyVZLSVtLSVkLSVILSVNLSVTJykKUEFUQ0hfTE9HPSIkbG9nRGlyL3BhdGNoLXZjZmFwcjE5MDUtJHRpbWVzdGFtcC5sb2ciCmV4cG9ydCBLVUJFQ09ORklHPS9ldGMva3ViZXJuZXRlcy9hZG1pbi5jb25mCgpmdW5jdGlvbiBsb2coKSB7CiAgbG9jYWwgbXNnPSIkMSIKICBsb2NhbCBsZXZlbD0iJDIiCiAgbG9jYWwgZHQ9JChkYXRlICcrJVktJW0tJWQgJUg6JU06JVMnKQogIGxvY2FsIGhvc3ROYW1lPSIkKGhvc3RuYW1lIC1mKSIKICBlY2hvICJbJGxldmVsXVskZHRdWyRob3N0TmFtZV0gJG1zZyIgPiYyCn0KCmZ1bmN0aW9uIGxvZ19pbmZvKCkgewogIGxvZyAiJDEiICJJTkZPIgp9CgpmdW5jdGlvbiBsb2dfZXJyb3IoKSB7CiAgbG9nICIkMSIgIkVSUk9SIgp9CgpmdW5jdGlvbiBkaWUoKSB7CiAgbG9nX2Vycm9yICIkMSIKICBleGl0IDEKfQoKbWtkaXIgLXAgIiRsb2dEaXIiIHx8IGRpZSAiRmFpbGVkIHRvIGNyZWF0ZSBsb2cgZGlyZWN0b3J5ICRsb2dEaXIuIgoKZXhlYyA+ID4odGVlIC1hICIkUEFUQ0hfTE9HIikgMj4mMQoKbG9nX2luZm8gIj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0iCmxvZ19pbmZvICJJbnN0YWxsaW5nIG1ldHJpY3Mtc2VydmVyIFJCQUMgZml4IChWQ0ZBUFItMTkwNSkuIgpsb2dfaW5mbyAiPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PSIKCiMgUGF0Y2hlcyB0aGUgb24tZGlzayBjaGFydCB0ZW1wbGF0ZS4gUnVucyBvbiBldmVyeSBjbHVzdGVyIG5vZGUgdmlhCiMgInZyYWNsaSBjbHVzdGVyIGV4ZWMiIGJlbG93LCBzaW5jZSBlYWNoIG5vZGUgaGFzIGl0cyBvd24gbG9jYWwgY29weSBvZgojIC9vcHQvY2hhcnRzIGFuZCBhbnkgb2YgdGhlbSBjb3VsZCBiZSB0aGUgb25lIHRoYXQgbmV4dCBydW5zICJ1cGdyYWRlLWs4cyIuCk5PREVfUEFUQ0hfU0NSSVBUPSQoY2F0IDw8ICdOT0RFX0VPRicKIyEvYmluL2Jhc2gKc2V0IC11byBwaXBlZmFpbAoKQ0hBUlRfRklMRT0iL29wdC9jaGFydHMvcHJlbHVkZS1rOHMvdGVtcGxhdGVzL21ldHJpY3Mtc2VydmVyLnlhbWwiCkJBQ0tVUF9ESVI9Ii9kYXRhL3BhdGNoLWJhY2t1cC92Y2ZhcHIxOTA1LSQoZGF0ZSAnKyVZLSVtLSVkLSVILSVNLSVTJykiCgpmdW5jdGlvbiBsb2coKSB7CiAgbG9jYWwgbXNnPSIkMSIKICBsb2NhbCBsZXZlbD0iJDIiCiAgbG9jYWwgZHQ9JChkYXRlICcrJVktJW0tJWQgJUg6JU06JVMnKQogIGxvY2FsIGhvc3ROYW1lPSIkKGhvc3RuYW1lIC1mKSIKICBlY2hvICJbJGxldmVsXVskZHRdWyRob3N0TmFtZV0gJG1zZyIgPiYyCn0KCmZ1bmN0aW9uIGxvZ19pbmZvKCkgewogIGxvZyAiJDEiICJJTkZPIgp9CgpmdW5jdGlvbiBsb2dfZXJyb3IoKSB7CiAgbG9nICIkMSIgIkVSUk9SIgp9CgpmdW5jdGlvbiBkaWUoKSB7CiAgbG9nX2Vycm9yICIkMSIKICBleGl0IDEKfQoKaWYgZ3JlcCAtcUUgJ15ccyotXHMqbm9kZXMvbWV0cmljc1xzKiQnICIkQ0hBUlRfRklMRSI7IHRoZW4KICBsb2dfaW5mbyAiJENIQVJUX0ZJTEUgYWxyZWFkeSBjb250YWlucyAnbm9kZXMvbWV0cmljcycgLSBjaGFydCB0ZW1wbGF0ZSBhbHJlYWR5IHBhdGNoZWQuIE5vdGhpbmcgdG8gZG8uIgogIGV4aXQgMApmaQoKZ3JlcCAtcUUgJ15ccyotXHMqbm9kZXMvc3RhdHNccyokJyAiJENIQVJUX0ZJTEUiIHx8IGRpZSAiJENIQVJUX0ZJTEUgZG9lcyBub3QgY29udGFpbiAnbm9kZXMvc3RhdHMnIG9yICdub2Rlcy9tZXRyaWNzJyAtIGNoYXJ0IGNvbnRlbnQgZGlmZmVycyBmcm9tIHdoYXQgdGhpcyBwYXRjaCBleHBlY3RzLiIKCm1rZGlyIC1wICIkQkFDS1VQX0RJUiIgfHwgZGllICJGYWlsZWQgdG8gY3JlYXRlIGJhY2t1cCBkaXJlY3RvcnkgJEJBQ0tVUF9ESVIuIgpjcCAiJENIQVJUX0ZJTEUiICIkQkFDS1VQX0RJUi9tZXRyaWNzLXNlcnZlci55YW1sIiB8fCBkaWUgIkZhaWxlZCB0byBiYWNrIHVwICRDSEFSVF9GSUxFIHRvICRCQUNLVVBfRElSL21ldHJpY3Mtc2VydmVyLnlhbWwuIgpsb2dfaW5mbyAiQmFja2VkIHVwICRDSEFSVF9GSUxFIHRvICRCQUNLVVBfRElSL21ldHJpY3Mtc2VydmVyLnlhbWwiCgpzZWQgLWkgLUUgJ3MjXihccyotXHMqKW5vZGVzL3N0YXRzXHMqJCNcMW5vZGVzL21ldHJpY3MjJyAiJENIQVJUX0ZJTEUiIHx8IGRpZSAiRmFpbGVkIHRvIHBhdGNoICRDSEFSVF9GSUxFLiBSZXN0b3JlIGl0IGZyb20gJEJBQ0tVUF9ESVIvbWV0cmljcy1zZXJ2ZXIueWFtbCBpZiBuZWVkZWQuIgpsb2dfaW5mbyAiUGF0Y2hlZCAkQ0hBUlRfRklMRTogbm9kZXMvc3RhdHMgLT4gbm9kZXMvbWV0cmljcyIKTk9ERV9FT0YKKQoKbG9nX2luZm8gIlBhdGNoaW5nIG9uLWRpc2sgY2hhcnQgdGVtcGxhdGUgb24gYWxsIGNsdXN0ZXIgbm9kZXMgLi4uIgpOT0RFX1BBVENIX1NDUklQVF9CNjQ9JChiYXNlNjQgLXcwIDw8PCAiJE5PREVfUEFUQ0hfU0NSSVBUIikKdnJhY2xpIGNsdXN0ZXIgZXhlYyAtLSBiYXNoIC1jICJlY2hvICROT0RFX1BBVENIX1NDUklQVF9CNjQgfCBiYXNlNjQgLWQgfCBiYXNoIiB8fCBkaWUgIlBhdGNoaW5nIHRoZSBvbi1kaXNrIGNoYXJ0IHRlbXBsYXRlIGZhaWxlZCBvbiBvbmUgb3IgbW9yZSBub2Rlcy4iCgpCQUNLVVBfRElSPSIvZGF0YS9wYXRjaC1iYWNrdXAvdmNmYXByMTkwNS1jbHVzdGVycm9sZS0kdGltZXN0YW1wIgpta2RpciAtcCAiJEJBQ0tVUF9ESVIiIHx8IGRpZSAiRmFpbGVkIHRvIGNyZWF0ZSBiYWNrdXAgZGlyZWN0b3J5ICRCQUNLVVBfRElSLiIKCmxvZ19pbmZvICJCYWNraW5nIHVwIHRoZSBsaXZlIHN5c3RlbTptZXRyaWNzLXNlcnZlciBDbHVzdGVyUm9sZSB0byAkQkFDS1VQX0RJUi9zeXN0ZW0tbWV0cmljcy1zZXJ2ZXItY2x1c3RlcnJvbGUueWFtbCAuLi4iCi91c3IvbG9jYWwvYmluL2t1YmVjdGwgZ2V0IGNsdXN0ZXJyb2xlIHN5c3RlbTptZXRyaWNzLXNlcnZlciAtbyB5YW1sID4gIiRCQUNLVVBfRElSL3N5c3RlbS1tZXRyaWNzLXNlcnZlci1jbHVzdGVycm9sZS55YW1sIiB8fCBkaWUgIkZhaWxlZCB0byBiYWNrIHVwIHRoZSBsaXZlIHN5c3RlbTptZXRyaWNzLXNlcnZlciBDbHVzdGVyUm9sZS4iCgpsb2dfaW5mbyAiQXBwbHlpbmcgdGhlIGZpeCB0byB0aGUgbGl2ZSBjbHVzdGVyIG5vdyAoUkJBQyB0YWtlcyBlZmZlY3QgaW1tZWRpYXRlbHksIG5vIHJlc3RhcnQgbmVlZGVkKSAuLi4iCi91c3IvbG9jYWwvYmluL2t1YmVjdGwgYXBwbHkgLWYgLSA8PCdFT0YnIHx8IGRpZSAiRmFpbGVkIHRvIGFwcGx5IHRoZSBsaXZlIENsdXN0ZXJSb2xlIGZpeC4iCmFwaVZlcnNpb246IHJiYWMuYXV0aG9yaXphdGlvbi5rOHMuaW8vdjEKa2luZDogQ2x1c3RlclJvbGUKbWV0YWRhdGE6CiAgbmFtZTogc3lzdGVtOm1ldHJpY3Mtc2VydmVyCnJ1bGVzOgotIGFwaUdyb3VwczoKICAtICIiCiAgcmVzb3VyY2VzOgogIC0gcG9kcwogIC0gbm9kZXMKICAtIG5vZGVzL21ldHJpY3MKICAtIG5hbWVzcGFjZXMKICAtIGNvbmZpZ21hcHMKICB2ZXJiczoKICAtIGdldAogIC0gbGlzdAogIC0gd2F0Y2gKRU9GCgpsb2dfaW5mbyAiV2FpdGluZyBmb3IgbWV0cmljcy1zZXJ2ZXIgdG8gcGljayB1cCB0aGUgZml4IGFuZCB2ZXJpZnlpbmcgazhzLW1ldHJpY3MgLi4uIgp2ZXJpZmllZD1mYWxzZQptYXhfYXR0ZW1wdHM9MTIKcmV0cnlfaW50ZXJ2YWw9MTAKZm9yICgoYXR0ZW1wdCA9IDE7IGF0dGVtcHQgPD0gbWF4X2F0dGVtcHRzOyBhdHRlbXB0KyspKTsgZG8KICBzbGVlcCAiJHJldHJ5X2ludGVydmFsIgogIG91dHB1dD0kKC91c3IvbG9jYWwvYmluL2s4cy1tZXRyaWNzIDI+JjEpCiAgaWYgZ3JlcCAtcSAiTm9kZSBOYW1lIiA8PDwgIiRvdXRwdXQiOyB0aGVuCiAgICB2ZXJpZmllZD10cnVlCiAgICBicmVhawogIGZpCiAgbG9nX2luZm8gIkF0dGVtcHQgJGF0dGVtcHQvJG1heF9hdHRlbXB0czogazhzLW1ldHJpY3Mgbm90IHJlcG9ydGluZyBkYXRhIHlldCwgcmV0cnlpbmcgLi4uIgpkb25lCgppZiBbICIkdmVyaWZpZWQiID0gdHJ1ZSBdOyB0aGVuCiAgbG9nX2luZm8gIlZlcmlmaWNhdGlvbiBzdWNjZWVkZWQ6IGs4cy1tZXRyaWNzIGlzIHJlcG9ydGluZyBkYXRhLiIKZWxzZQogIGxvZ19pbmZvICJQYXRjaCBhcHBsaWVkIHN1Y2Nlc3NmdWxseSBidXQgdW5hYmxlIHRvIHZlcmlmeSBrOHMtbWV0cmljcyBpcyB3b3JraW5nIHlldC4gSXQgbWF5IG5lZWQgbW9yZSB0aW1lIHRvIHN0YXJ0IHJlcG9ydGluZyBkYXRhLiIKZmkKCmxvZ19pbmZvICJQYXRjaGluZyBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5LiIK | bash -

3. The log is written to /var/log/vmware/prelude/patch-vcfapr1905-<timestamp>.log on the node the command was run on. 

=============================================
Rollback patch
=============================================

The patch backs up the original chart template on each node at:

/data/patch-backup/vcfapr1905-<per-node-timestamp>/metrics-server.yaml

and the original live ClusterRole, on the node the command was run on, in a separately named directory (so the two backups aren't mistaken for each other on that node) at:

/data/patch-backup/vcfapr1905-clusterrole-<timestamp>/system-metrics-server-clusterrole.yaml

(same <timestamp> as that node's /var/log/vmware/prelude/patch-vcfapr1905-<timestamp>.log)

To roll back on all nodes, first find each node's backup timestamp, then restore (substitute the actual timestamp found on each node):

vracli cluster exec -- bash -c 'ls -d /data/patch-backup/vcfapr1905-[0-9]* 2>/dev/null | tail -1'
vracli cluster exec -- bash -c 'cp /data/patch-backup/vcfapr1905-<timestamp>/metrics-server.yaml /opt/charts/prelude-k8s/templates/metrics-server.yaml'

Then revert the live ClusterRole from the node the patch was run on, using its backup:

kubectl apply -f /data/patch-backup/vcfapr1905-clusterrole-<timestamp>/system-metrics-server-clusterrole.yaml

=============================================
Verification
=============================================

Run k8s-metrics on any node - it should print a table of per-node CPU/memory usage followed by a table of per-pod usage, with no jq errors:

/usr/local/bin/k8s-metrics

Additional Information

If the issue persists, see Contact Broadcom Support for further assistance.