All Control Compliance Suite data collection jobs are hanging in pending status and are never delivered by the Load Balancer to a data collecting CCS Manager

Products

• Control Compliance Suite Windows
• Control Compliance Suite
• Control Compliance Suite Standards Server

Issue/Introduction

All data collection jobs are hanging in PENDING mode when viewed in the Data Collection Status window in the Control Compliance Suite (CCS) console.  Log analysis of Load Balancer CCS Manager shows job is not getting routed to the appropriate data collecting CCS Managers.

Jobs eventually complete with no data returned and no errors in the console.

 

Examples of significant errors in CCS Manager Load Balancer role logs:

19-01-04 15:34:16.119,2019-01-04 10:34:16.119,<CCSM_hostname>,Error,MOS_Types,Symantec.CSM.DPS,190312,Dispatcher Submit Queue:100,100,GetPlatformDetails,,0,0,[WindowsPlatformEntity.WindowsPlatformEntity.SystemID] not found.

2019-01-04 15:34:16.119,2019-01-04 10:34:16.119,<CCSM_hostname>,Error,MOS_Types,Symantec.CSM.DPS,190312,Dispatcher Submit Queue:100,100,GetPlatformDetails,,0,0,"System.Collections.Generic.KeyNotFoundException: [WindowsPlatformEntity.WindowsPlatformEntity.SystemID] not found.

   at Symantec.MOS.Types.InstanceCache.GetPlatformDetails(String fullyQualifiedMosObjectName, Boolean bThrow)

   at Symantec.MOS.Types.InstanceCache.LoadPlatformDetails(String fullyQualifiedMosObjectName, Boolean bThrow)

   at Symantec.MOS.Types.InstanceCache.Lookup(String fullyQualifiedMosObjectName, Type typeExpected, Boolean bThrow)

   at Symantec.MOS.Types.InstanceCache.LookupField(String fieldName, Boolean bthrow)"

.

.

2019-01-04 15:18:09.069,2019-01-04 10:18:09.069,<CCSM_hostname>,Error,Symantec.CSM.DPS,Symantec.CSM.DPS,190312,,9,EnsureThereAreApplicationEndpoints,,0,0,"Error while setting routing. : System.InvalidOperationException: Service 'System.ServiceModel.Routing.RoutingService' has zero application (non-infrastructure) endpoints. This might be because no configuration file was found for your application, or because no service element matching the service name could be found in the configuration file, or because no endpoints were defined in the service element.

   at System.ServiceModel.Description.DispatcherBuilder.EnsureThereAreApplicationEndpoints(ServiceDescription description)

   at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)

   at System.ServiceModel.ServiceHostBase.InitializeRuntime()

   at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)

   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

   at Symantec.Blade.Server.ServiceExecutor.StartDPSHopping(X509Certificate2 serverCertificate, String strServerIndentity)"

Environment

CCS 12.x.x managers.  (

 

Cause

Missing custom platform field XML files on the CCS Manager in load balancer role. 

 

Resolution

The issue was due to the CCS install having extended custom fields added to the platform schema(s).  When custom platform fields are added, special schema XML files for those fields are added to the root of the DPS folder system on the CCS Manager machine.

During a data collection job run, the CCS Manager receives the job information from the Application Server which has the custom schema XML files referenced.  The Load Balancer CCS Manager then tries to reference the fields and fails.  Because of this it cannot make a routing decision.

All CCS Managers, as well as the application server, should have copies of the identical XML custom schema files. These files do not have consistent names.

To fix this issue:

      Copy these files from where they exist to the Load Balancer CCS Manager's <install path>\Program Files (x86)\CCS\Reporting and Analytics\DPS root folder.  Restart the Symantec Data Processing service. Data collection jobs should now route according to the routing rules in place and be delivered to data collecting CCS Managers.  NOTE: These files should be copied to all of your CCS managers (i.e Managers with Evaluation Role, etc) as this issue can occur on any manager and not just managers that are the Load Balancer.

IMPORTANT NOTE:

       Before uninstalling a CCS Manager (with intent to reinstall) check to ensure that your assets do not have custom fields showing in their CUSTOM tab in the Asset's properties screen.  If they do, ensure that you first copy all custom schema XML files from the CCS Manager's <install path>\Program Files (x86)\CCS\Reporting and Analytics\DPS  first.  These can then be restored after the CCS Manager is reinstalled.