AIOps - Installer fails with [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster.

Products

• CA Application Performance Management (APM / Wily / Introscope)
• DX Operational Intelligence
• CA App Experience Analytics

Issue/Introduction

We have configured the Elastic Node as per documentation and show below:

kubectl get nodes --show-labels

..

Ready    node     28d   v1.21.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dxi-es-node=master-data-1,kubernetes.io/arch=amd64,kubernetes.io/hostname=<your-ELASTIC-servername>,kubernetes.io/os=linux,node-role.kubernetes.io/node=
...

However DX Platform fails to recognize the node as available. Why?

...
2021-06-14 13:00:57,169 INFO  - [image-prefix-validate}] [ OK ] Image prefix has been successfully validated.
2021-06-14 13:01:00,735 TRACE - POST: [email protected]
2021-06-14 13:01:02,106 TRACE - POST: [email protected]
2021-06-14 13:01:02,132 TRACE - POST: [email protected]
2021-06-14 13:01:02,261 INFO  - [registry-validate] [ OK ] Registry has been validated successfully
2021-06-14 13:01:02,265 TRACE - DELETE: [email protected]
2021-06-14 13:01:02,311 INFO  - [node-info] [ INFO ] Node <your-ELASTIC-servername> is unschedulable
...

2021-06-14 13:01:02,323 INFO  - [environment-validate] [ERROR] Environment Configuration verification failed.
2021-06-14 13:01:06,288 INFO  - [nginx-version] [ OK ] Nginx Ingress Controller version has been verified successfully
2021-06-14 13:01:22,192 INFO  - [node-count] Ready and schedulable nodes with dxi-es-node label count equals 0.
2021-06-14 13:01:22,192 INFO  - [node-count] Minimum one ready and schedulable node required with 'dxi-es-node' label.
2021-06-14 13:01:22,192 ERROR - [elasticsearch-node-validate] [ERROR] No nodes were detected in the cluster. 

Environment

DX Operational Intelligence 20.x
DX Application Performance Management 20.x
DX AXA 20.x

Cause

From "kubectl describe nodes" output:

...
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity: <your-ELASTIC-servername>
..

The above output indicates that the node is set to unschedulable which prevents the scheduler from placing new pods onto that Node hence the installer consider the elastic node as unavailable.

Resolution

Untaint the Elastic server node(s) using the below command:

kubectl taint node <ELASTIC servername> node-role.kubernetes.io/master-

Additional Information

DX AIOPs Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Other external references:
https://stackoverflow.com/questions/55191980/remove-node-role-kubernetes-io-masternoschedule-taint
https://stackoverflow.com/questions/57803471/in-k8s-how-to-setup-to-demo-kubectl-taint-nodes-all-node-role-kubernetes-io-m