CCS jobs failing with error 'There are no CCS Manager Load Balancers available for job execution'.
search cancel

CCS jobs failing with error 'There are no CCS Manager Load Balancers available for job execution'.

book

Article ID: 212292

calendar_today

Updated On:

Products

Control Compliance Suite Standards Database Control Compliance Suite Control Compliance Suite Standards Server

Issue/Introduction

When running jobs, they are failing with the following error:

In the console you may see the following error:

Object reference not set to an instance of an object.

In the AppServer logs you may see something similar to the following:

Symantec.Interfaces.AppServer.JobManagement.DataCollectionException: An error occurred in Data Collection activity: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.NullReferenceException: Object reference not set to an instance of an object.

Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution.

or

An error occurred in Data Query activity: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

 

NOTE: These errors can be reported when running a Data Collection job, a CER job, or even a Fetch Registered Agents job.

Environment

CCS 12.6.x - 12.7

Cause

The blob the credentials are stored in (specifically the passwords) has become corrupted.  Look in the AppServer*.log file and search for an error similar to this.

An error occurred in Data Collection activity: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

An error occurred in Data Query activity: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

This may result in the following entry in the log file as well, causing the "Object reference not set to an instance of an object." in the console for failed jobs.

Symantec.Interfaces.AppServer.JobManagement.DataCollectionException: An error occurred in Data Collection activity: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Error in submitting query job to CCS Manager: There are no CCS Manager Load Balancers available for job execution. ---> Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.NullReferenceException: Object reference not set to an instance of an object.

 

This can also happen on Command Based checks:

2022-08-13 06:51:38.487,2022-08-12 23:51:38.487,######,Warning,Standards,AppserverService,4632,,61,UpdateCommandInfoForCommandChecks,,0,1,Command info not specified for the command based check - Name:'5.7 Ensure access to the su command is restricted' Version:'1.0.0' ID:'50026545-####-####-####-e024fd01c2fe'

2022-08-13 06:51:45.805,2022-08-12 23:51:45.805,######,Error,Symantec.CSM.EncryptionManagement,AppserverService,4632,,61,RequestBlobDecryption,,0,1,"Encrypt.RequestBlobDecryption() failed : System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

2022-08-13 06:51:45.868,2022-08-12 23:51:45.868,######,Error,QueryParamsJobAttachmentManager,AppserverService,4632,,61,RequestBlobDecryption,,0,1,"Error while attaching attachment-based check query params. : System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
2022-08-13 06:51:45.977,2022-08-12 23:51:45.977,######,Error,QueryAdditionalParamManager,AppserverService,4632,,61,RequestBlobDecryption,,0,0,"Error while attaching query params to request : System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
2022-08-13 06:51:45.977,2022-08-12 23:51:45.977,######,Error,StandardsModule.Activities,AppserverService,4632,,61,RequestBlobDecryption,,0,0,"Error creating blade job definition in BladeRunnerQueryJob.GetRequestObject() : System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
2022-08-13 06:51:45.977,2022-08-12 23:51:45.977,######,Error,AppServer.BladeRunnerJobs,AppserverService,4632,,61,Route,,0,0,"Failed to submit job to DPS Load Balancer ######:5600 spn=CCSManager-###### Exception: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
2022-08-13 06:51:45.977,2022-08-12 23:51:45.977,######,Error,AppServer.WorkFlowSvc,AppserverService,4632,,61,RequestBlobDecryption,,0,0,"Error in BladeJobService.SubmitBladeRunnerJob. : Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.
2022-08-13 06:51:45.977,2022-08-12 23:51:45.977,######,Error,AppServer.Activities,AppserverService,4632,,61,RequestBlobDecryption,,0,0,"Error in submitting job to blade for dataCollectionActivity with AdamID=14c5de8c-####-####-####-c5384b563495, InstanceID=a9d6be38-####-####-####-98c8f729c976. There are no CCS Manager Load Balancers available for job execution. : Symantec.CSM.AppServer.BladeRunnerJobs.RoutingException: There are no CCS Manager Load Balancers available for job execution. ---> System.ApplicationException: Blob decryption failed: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.

Resolution

To correct this issue do the following steps for the issue with Credentials.

  1. In the CCS Console, navigate over to the Admin -> Credentials workspace.
  2. Make note of all the credentials listed and how they are configured, such as any assets for asset-specific credentials. So, they can be recreated later on.
  3. Delete ALL* of the credentials listed, or this workaround won't work. (You will need to reboot the Application Server after this step)
  4. Recreate all the needed credentials.
  5. Re-run any jobs that were previously failing, they should now run without any issues.

 

To correct this issue do the following steps for the issue with Command Based Checks, for predefined standards.

  1. In the CCS Console, find the predefined standard being used, and delete it.
  2. Close the CCS Console, and on the Application Server open the Control Panel.
  3. Open Programs and Features/Uninstall a Program and select Symantec Control Compliance Suite 12.<version>
  4. Then select Uninstall/Change, and after the content install launches select Add/Upgrade.
  5. Uncheck anything you do not want to be installed, and make sure the previously deleted standards are selected.
  6. Go through the different windows selecting Next, until you can Install.
  7. Once finished close, and relaunch CCS Console
  8. You will now be able to use the newly installed predefined standards to run the job successfully.

NOTE: In the case of a custom standard/check you will need to either copy from a known good source, or manually recreate the check.

For additional details on how this can happen please refer to the following documentation.
About passphrases for Application Server Service and Encryption Management Service

Additional Information

NOTE: When modifying the Application Server Credentials, you may run into the following error.

Failed to create JobManager Secure storage instance.

If you see that error, then make sure you followed the resolution, and after step 3, reboot the server.
Then go back, and you will now be able to add in the credential/password to create the Secure storage.

*From step 3 from above under "To correct this issue do the following steps for the issue with Credentials."
Connect to the production database (CSM_DB) and run the following query, verify that nothing is returned. If anything is returned it will tell you which type of credential still exists, select that type in the console and delete all the credentials listed.

USE CSM_DB
GO

SELECT ValueAsString AS [CredentialType] FROM [CSM_DB].[dbo].[Credentials]
WHERE PropertyName = 'internal:CredentialType'

Powered by Wolken Software