Control Compliance Suite (CCS) jobs failing with error there are no ccs manager load balancers available for job execution.

Products

• Control Compliance Suite Standards Database
• Control Compliance Suite
• Control Compliance Suite Standards Server
• Control Compliance Suite Windows
• Control Compliance Suite Unix

Issue/Introduction

When running jobs, they are failing with the following error:

Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution.

OR

An error occurred in Data Query activity: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

Cause

The blob the credentials are stored in (specifically the passwords) has become corrupted.  Look in the AppServer*.log file and search for an error similar to this.

An error occurred in Data Collection activity: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

OR

An error occurred in Data Query activity: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
  at System.Security.Cryptography.CapiSymmetricAlgorithm.DepadBlock(Byte[] block, Int32 offset, Int32 count)
  at System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[] inutBuffer, Int32 inputOffset, Int32 inputCount)
  at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
  at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
  at System.IO.Stream.Close()
  at Symantec.CSM.SecureStorage.Cryptography.Crypt(Byte[] blob, Byte[] key, CryptoStreamMode mode)
  at Symantec.CSM.EncryptionManagement.Server.EncryptionManagementServer.ReqDecryptBlobHandler(DispatchObject obj)

Resolution

To correct this issue do the following steps.

1. In the CCS Console, navigate over to the Credentials workspace.
2. Take a note of all the credentials listed and how they are configured, such as any assets for asset specific credentials.
3. Delete ALL of the credentials listed, or this won't work. (May need to reboot the AppServer after this step)
4. Recreate all the needed credentials.
5. Re-run any jobs that were previously failing, they should now run without any issue.

Additional Information

NOTE: When modifying the Application Server Credentials, you may run into the following error.

Failed to create JobManager Secure storage intance.

If you see that error, then make sure you followed the resolution, after step 3, reboot the server.
Then go back, you will now be able to add in the credential/password to create the Secure storage.