AIOps - Installation fails with [ ERROR ] Pod of Deployment .. has failed to start in the given timeframe

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

Installer fails with below error messages:

...

[verify] [ ERROR ] Verification pods of Deployment ng-acc-configserver-deployment: [Running]
[verify] [ ERROR ] Pod ng-acc-configserver-deployment-fdc765857-hjdwn Initialized: True -> null
[verify] [ ERROR ] Pod ng-acc-configserver-deployment-fdc765857-hjdwn Ready: False -> containers with unready status: [ng-acc-configserver-container]
[verify] [ ERROR ] Pod ng-acc-configserver-deployment-fdc765857-hjdwn ContainersReady: False -> containers with unready status: [ng-acc-configserver-container]
[verify] [ ERROR ] Pod ng-acc-configserver-deployment-fdc765857-hjdwn PodScheduled: True -> null
[verify] [ ERROR ] Pod of Deployment ng-acc-configserver-deployment has failed to start in the given timeframe.

..

[verify] [ ERROR ] Pod ng-acc-repository-deployment-6fb69885f8-d8kkj Initialized: False -> containers with incomplete status: [init-ds]
[verify] [ ERROR ] Pod ng-acc-repository-deployment-6fb69885f8-d8kkj Ready: False -> containers with unready status: [ng-acc-repository-bundles-container ng-acc-repository-container]
[verify] [ ERROR ] Pod ng-acc-repository-deployment-6fb69885f8-d8kkj ContainersReady: False -> containers with unready status: [ng-acc-repository-bundles-container ng-acc-repository-container]
[verify] [ ERROR ] Pod ng-acc-repository-deployment-6fb69885f8-d8kkj PodScheduled: True -> null
[verify] [ ERROR ] Pod of Deployment ng-acc-repository-deployment has failed to start in the given timeframe.

 

Environment

DX Platform Installer 20.2.x

Cause

-Poor NFS Throughput (IOps and speed)
-NFS not using SSD as per hardware recommendation
-Poor Network Speed (recommended NIC 10 GB/s between the nodes)

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2/installing/software-requirements.html#concept.dita_d7ab03039be1dd1c0bbba340c2848b6d7d6349f2_HardwareRequirements

Resolution

1) Use dbench-fio.sh script to validate NFS Throughput

https://knowledge.broadcom.com/external/article/212399

2) Make sure your setup aligns with all the hardware recommendations and prerequisites:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-platform-on-premise/20-2/installing/software-requirements.html#concept.dita_d7ab03039be1dd1c0bbba340c2848b6d7d6349f2_HardwareRequirements

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815