PAM Not starting up, Error Error finalising cipher data: pad block corrupted
Article ID: 220086
Updated On:
Products
Issue/Introduction
Silently installed pam 4.3.05 and patched with the latest CP, on start up it shows the attached error.
...
<ERROR 2021-07-09 11:34:39,104 BaseLogger::error> OasisConfig does not exist meaning fresh installation, So no need to check
...
<INFO 2021-07-09 11:36:19,203 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO 2021-07-09 11:36:19,215 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO 2021-07-09 11:36:19,222 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO 2021-07-09 11:36:19,229 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
...
<ERROR 2021-07-09 11:41:10,024 SimpleCastorXMLUtil::error> Mapping File serviceOperationMappingFilenot Found. This may be available at GUI or Source.
<ERROR 2021-07-09 11:41:10,024 SimpleCastorXMLUtil::error> SimpleCastorXMLUtil.getMapping: failed to find mapping serviceOperationMappingFile file: serviceOperationMappingFile
...
<ERROR 2021-07-09 11:42:03,212 MirrorDomain::error> Exception while processing mirrored file https://<pamserver>:443/mirroringrepository/.c2oserverresources/lib/jackson-databind-2.9.10.2.jar as local file D:\Program Files\CA\PAM\server\c2o\.c2orepository\.c2oserverresources\lib\jackson-databind-2.9.10.2.jar
java.io.FileNotFoundException: https://<pamserver>:443/mirroringrepository/.c2oserverresources/lib/jackson-databind-2.9.10.2.jar
...
<INFO 2021-07-09 11:44:27,024 OasisNodeInstaller::createFile> error creating file: D:\Program Files\CA\PAM\server\c2o\bin\wrapper.exe (The process cannot access the file because it is being used by another process)
<INFO 2021-07-09 11:44:27,024 OasisNodeInstaller::createNewFile> creating file: D:\Program Files\CA\PAM\server\c2o\bin\c2osvcw.conf
<WARN 2021-07-09 11:44:27,056 FileUtil::warn> Reading comment properties from oasisconfigComment.properties are done
...
<ERROR 2021-07-09 11:47:19,304 SimpleCastorXMLUtil::error> Mapping File serviceOperationMappingFilenot Found. This may be available at GUI or Source.
<ERROR 2021-07-09 11:47:19,304 SimpleCastorXMLUtil::error> SimpleCastorXMLUtil.getMapping: failed to find mapping serviceOperationMappingFile file: serviceOperationMappingFile
...
Thirdparty files not copying to secondary nodes.
Properties table filling up with entries
Environment
Release : 4.3
Component : Process Automation
Resolution
The root cause is, as we are repaving the application from windows to the application level, it's installed as a fresh installation rather than an upgrade. That's the reason, it's creating new entries and the properties table is growing.
Having duplicate server IDs in the properties table can impact the process execution and it's critical to address.
Before starting the newly installed PAM node, we need to clean its old references so that it can insert the new entries automatically during the bootup.
How to achieve this?
1) Query to find the cluster node references
Get the cluster node ID:
SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')
Here "TESTSERVER1" is the sample PAM hostname and we need the first 35 characters which is the cluster node ID
Get all the references of Cluster Node:
SELECT * from properties where propkey like ('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')+'%')
2) Query to delete cluster node the references
delete from properties where propkey like ('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')+'%')
Everything should be automated and no human intervention. So, please find the Sample PowerShell to automate it.
-------------------------------------------------------------------------------------------------------------------------------------------------
$pamHostName="TESTSERVER1"
$queryString="delete from properties where propkey like "+
"('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties "+
"where propvalue='"+$pamHostName+"[7900]')+'%')"
Invoke-Sqlcmd -Query $queryString -ServerInstance "DBSERVER\SQL02" -Database "PAM_435_CP04_gMSA_Run"
-------------------------------------------------------------------------------------------------------------------------------------------------
Please ensure to execute this before starting the repaved PAM node.
Feedback
