PAM Not starting up, Error Error finalising cipher data: pad block corrupted
search cancel

PAM Not starting up, Error Error finalising cipher data: pad block corrupted

book

Article ID: 220086

calendar_today

Updated On:

Products

CA Process Automation Base

Issue/Introduction

I've silently installed pam 4/3/05 and patched with the latest CP, on start up it shows the attached error.

...
<ERROR 2021-07-09 11:34:39,104 BaseLogger::error> OasisConfig does not exist meaning fresh installation, So no need to check
...
<INFO  2021-07-09 11:36:19,203 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO  2021-07-09 11:36:19,215 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO  2021-07-09 11:36:19,222 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
<INFO  2021-07-09 11:36:19,229 FIPSUtil::info> Decrypt using DES failed javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted Decrypting using AES
...
<ERROR 2021-07-09 11:41:10,024 SimpleCastorXMLUtil::error> Mapping File serviceOperationMappingFilenot Found. This may be available at GUI or Source.
<ERROR 2021-07-09 11:41:10,024 SimpleCastorXMLUtil::error> SimpleCastorXMLUtil.getMapping: failed to find mapping serviceOperationMappingFile file: serviceOperationMappingFile
...
<ERROR 2021-07-09 11:42:03,212 MirrorDomain::error> Exception while processing mirrored file https://<pamserver>:443/mirroringrepository/.c2oserverresources/lib/jackson-databind-2.9.10.2.jar as local file D:\Program Files\CA\PAM\server\c2o\.c2orepository\.c2oserverresources\lib\jackson-databind-2.9.10.2.jar
java.io.FileNotFoundException: https://<pamserver>:443/mirroringrepository/.c2oserverresources/lib/jackson-databind-2.9.10.2.jar
...
<INFO  2021-07-09 11:44:27,024 OasisNodeInstaller::createFile> error creating file: D:\Program Files\CA\PAM\server\c2o\bin\wrapper.exe (The process cannot access the file because it is being used by another process)
<INFO  2021-07-09 11:44:27,024 OasisNodeInstaller::createNewFile> creating file: D:\Program Files\CA\PAM\server\c2o\bin\c2osvcw.conf
<WARN  2021-07-09 11:44:27,056 FileUtil::warn> Reading comment properties from oasisconfigComment.properties are done
...
<ERROR 2021-07-09 11:47:19,304 SimpleCastorXMLUtil::error> Mapping File serviceOperationMappingFilenot Found. This may be available at GUI or Source.
<ERROR 2021-07-09 11:47:19,304 SimpleCastorXMLUtil::error> SimpleCastorXMLUtil.getMapping: failed to find mapping serviceOperationMappingFile file: serviceOperationMappingFile
...

Thirdparty files not copying to secondary nodes.
properties table filling up with entries

Environment

Release : 4.3

Component : Process Automation

Resolution

The root cause is, as we are repaving the application from windows to the application level, it's installed as a fresh installation rather than an upgrade.  That's the reason, it's creating new entries and the properties table is growing. 


Having duplicate server IDs in the properties table can impact the process execution and it's critical to address. 
It's not an issue from an application perspective but we need a solution for DC migration to successfully repave. 
 
So, Please find the proposed solution.
Before starting the newly installed PAM node, we need to clean its old references so that it can insert the new entries automatically during the bootup.


How to achieve this?
1) Query to find the cluster node references
Get the cluster node ID:
SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')
Here "TESTSERVER1" is the sample PAM hostname and we need the first 35 characters which is the cluster node ID


Get all the references of Cluster Node:
SELECT * from properties where propkey like ('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')+'%')


2) Query to delete cluster node the references
delete from properties where propkey like ('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties where propvalue='TESTSERVER1[7900]')+'%')

Everything should be automated and no human intervention. So, please find the Sample PowerShell to automate it. 
-------------------------------------------------------------------------------------------------------------------------------------------------
$pamHostName="TESTSERVER1"
$queryString="delete from properties where propkey like "+
"('%'+(SELECT SUBSTRING(propkey, 1, 34) AS ClusterNodeID from properties "+
"where propvalue='"+$pamHostName+"[7900]')+'%')"
Invoke-Sqlcmd -Query $queryString -ServerInstance "DBSERVER\SQL02" -Database "PAM_435_CP04_gMSA_Run"
-------------------------------------------------------------------------------------------------------------------------------------------------
Please ensure to execute this before starting the repaved PAM node.