After enabling secure ICAP or enablig SSL in REST API on the Protection Engine server, the service would not start. The SymantecProtectionEngineAbort.log contained the following information.
2022/08/29-11:11:54 1300 Error validating SSL/TLS configuration : Failed to add private key found in file /opt/SYMCScan/ssl.crt/ikey.jks to ctx. Error code: error:0909006C:PEM routines:get_name:no start line.
2022/08/29-11:11:54 0 Symantec Protection Engine is shutting down; logs may contain more information.
The certificates that were used in the configuration.xml file were not in PEM format. Protection Engine cannot read certificates or key files in p7b or any binary format. They must be BASE64 encoded so when opened in a text editor, the file contains ----BEGIN CERTIFICATE.
Ran the following to generate new certificates
openssl req -newkey rsa:4096 -nodes -x509 -days 3650 -keyout ikey.pem -out icert.pem
Updated the configuration.xml file PrivateKeyFile to point at the ikey.pem file
xmlmodifier -s /configuration/ProtocolSettings/ICAP/SecureICAP/PrivateKeyFile/@value <path to ikey.PEM File> configuration.xml
Updated the configuration.xml file CertificateFile to point at the icert.pem file
xmlmodifier -s /configuration/ProtocolSettings/ICAP/SecureICAP/CertificateFile/@value <path to icert.pem file> configuration.xml