High CPU Usage on all AP servers following r11.8 upgrade

Products

• Automation Point

Issue/Introduction

A week after we have started upgrades to 11.8 on the first subset of production servers with live consoles, we have just received maxed out CPU incidents and alerts for every single aforementioned server, all with in a couple hours of each other.  When we installed 11.8, each server was rebooted afterwards.  We never had any issue like this with 11.7.3.1.

Consoles are still accessible, but logging on is slow.  Remote connecting is slow.  CPU is constantly maxed.  Any action on the server is slow.

Each server is WINDOWS 2022 / 2 vCPU X 8 GB. The D:\ drives AP is installed is only 50GB, but 40GB are free.  There is no issue with the drive write speed.  The servers vary with how many consoles they have, some only host 10 sessions in the definition sets.

We are currently collecting an APDiag on each affected server.  It is taking a long time, 60+ minutes for each server to complete the task with a "(Not Responding)" status the whole time.  

 
Rebooting the servers fixes the CPU usage temporarily and then it reoccurs.

Environment

Automation Point 11.8 

Resolution

The associated APDiag was fairly unremarkable other than frequent reconnect of web services.  This was apparently happening with earlier release of AP as well, but the new OpenSSL3 loaded the CPU with new encryption, which is leading to the high CPU condition.   

To resolve the condition, disable the SSL by going to Expert Interface > Infrastructure > Web Services > Advanced and disable both SSL checkboxes

Since the communications happens over localhost, anyone listening would already have access.  

Additional Information

If this does not help to resolve the high CPU condition, change the Security level to be traced to 4 in the Expert Interface > Infrastructure > Error Tracing.   When the condition reoccurs, open a Support Case under product Automation Point and send in the APDiag for examination.