Recommended OS patch cycle for Performance Management environment

Products

• CA Performance Management
• Network Observability

Issue/Introduction

The Linux platform team wants setup the Performance Management servers to all be patched monthly.

What are the Best Practices, concern points and how to stage the patching in a fashion that’ll minimize data loss?

Environment

All supported Performance Management releases

Resolution

1. Ensure we've got valid backups for all DBs.
   1. PC netqosporal and em MySql DBs
   2. Data Repository Vertica DB
2. Follow proper stop/start procedures. Recommended path would be:
   1. Stop DA.Start patching.
   2. Stop DR DB. Start patching.
   3. Once DA and DR systems are patched ensure DB is restarted and then start DA.
   4. Stop, patch and restart one DC at a time or do them in batches. The data gap resulting from the patch cycle will be dependent on the length of time the DC is down.
   5. Stop PC services, patch, restart services.
3. Validate functionality.

Additional Information

Notes/Suggestions:

• Want a true/proper system backup for recovery should something go wrong on the host during/after patching? Follow these docs for full backups of PC, DA and DR.
  • PC:back-up-performance-center
  • DA: back-up-data-aggregator
  • DR:back-up-the-data-repository
• This KB article can be shared with others if we've got to rely on them to stop/start services before/after patching, or to confirm services are running post restart.
  
  • https://knowledge.broadcom.com/external/article?articleId=32043
• PC patching can be done first, last or in the middle. Really doesn't matter to much as it's function in this scenario is dependent on the up/down state of the DA, DR and DCs during the patching cycle.
• DC's should be left running during DA/DR patching in hopes they those go swiftly enough.
  • Once DA and DR are restarted and functional again, the DCs reconnect and feed their data back in.
  • Waiting a short time and validate in PC that data is current again, that it's fully caught up, before shutting down the DCs. If the DCs aren't done loading data back to DR via DA post DA/DR outage, when restarted data not yet loaded is going to be lost.