Not able to run Agent Product Update in CCS 12.6.1

Products

Control Compliance Suite Standards Server Control Compliance Suite Control Compliance Suite Standards Module

Issue/Introduction

When trying to upgrade a v12.5.2 (12.50.10200.1790) agent to the new v12.6 agent it gets an error.

Same thing for 12.5.1 (12.50.10128.1060) agents.

Failed to start the remote update process on agent.

Update Details:
Verifying signature of the package files... Done
Done
Connecting to agent server on the agent... Stopping all jobs on the agent before beginning update operation...Done
Closing connection with the agent update server... Done
Opening connection with the agent... Done
Connecting to update server on the agent... Done
Checking if manager is installed on the agent... Done
Checking if agent is running any other updates... Done
Checking for free space on agent... Done
Stopping the update support service if it is running on the agent... Done
Attempt no. 1 : Upload and run the support service on the agent...
Attempt no. 2 : Upload and run the support service on the agent...
Attempt no. 3 : Upload and run the support service on the agent...
Failed to install updates on agent '<agent_name>'
Error uploading and running update program on agent '<agent_name>'
Error copying C:\Program Files (x86)\Symantec\CCS\Reporting and Analytics\DPS\AgentUpdate\PU\apu-12.60\windows\smime\esmupdat.exe to <agent_name>
error writing request to update server on <agent_name>
error writing data header to remote process
An established connection was aborted by the software in your host machine

From the agent side, you would see something like this.

[07692] [2022/11/09 07:48:07:066] [INFO] Update server: Stopping one process: Esmupdate.
[07692] [2022/11/09 07:48:07:066] [NOTICE] Update server: Service 'Esmupdate' does not exists, hence no stop service required
[07692] [2022/11/09 07:48:12:079] [INFO] Update server: copy binary file esmupdat.exe
[07692] [2022/11/09 07:48:13:229] [ERROR] Update server: error:2107C080:PKCS7 routines:PKCS7_get0_signers:signer certificate not found
[07692] [2022/11/09 07:48:13:230] [ERROR] Update server: Signature verification failed for file `C:\Program Files (x86)\Symantec\CCS\Reporting and Analytics\RU\Package\esmupdat.exe', error-code=-8

Environment

Release: CCS 12.6.1

Cause

This is known issue and expected. CCS 12.6.1 APU has prerequisites for the agents to have a Hot Fix.

Due to this issue, it has updated certificates that are not compatible with older agents that have not had the Hot Fix for the new certificates applied.

Resolution

2 Hot Fixes were released 1 for 12.5.x agents and 1 for 12.6.0 agents.

Control Compliance Suite Agents Certificate Update Notification

As per the 12.6.1 documentation:

Installing or upgrading to CCS 12.6.1

Upgrading the CCS Agents

You can upgrade CCS Agents using the installer to version 12.6.1 from versions 12.5.1, 12.5.2, or 12.6.0. The procedure to upgrade the CCS Windows or CCS UNIX Agents to CCS 12.6.1 manually is the same as installing CCS Agents.

You can also upgrade the CCS Agents using the Agent Product Update job to CCS 12.6.1 from versions 12.5.1 onwards which includes the Hotfix 10003 for 12.6 agents or Hotfix 10205 for 12.5.x agents.

Upgrade CCS Agents manually using the installer

See Installing Control Compliance Suite Agent on Windows

See Installing CCS agent on UNIX

Upgrade CCS Agents using Agent Product Update

NOTE: Hotfix 10003 or 10205 is a pre-requisite for upgrading CCS Agents to 12.6.1 using APU. For more information, see Control Compliance Suite Agents Certificate Update Notification.

See Upgrading CCS Agents using Agent Product Update Job

Note that you can work around this.

Backup the current 12.6.1 APU on the AppServer "C:\ProgramData\Symantec\CCS\LiveUpdateStaging\CCS_AGENT_PU"
Move the "CCS1260.12_60_10100_1500_CCS_AGENT_PU" to a known safe location.
Take the 12.6.0 HF zip file extract out the APU zip and extract out the APU OR 12.5.2 HF
Copy the "CCS1260.12_60_10003_1300_CCS_AGENT_PU" folder OR "CCS1250.12_50_10205_1790_CCS_AGENT_PU" to the "C:\ProgramData\Symantec\CCS\LiveUpdateStaging\CCS_AGENT_PU"
Now in the CCS Console navigate to the LiveUpdate workspace, from the "Common Tasks" drop down menu select "Check Updates". Wait until this job finishes and shows the new Agent APU version.
We now need to finish clean up the 12.6.1 APU files. On the Application server navigate to "<InstallDir>\Application Server\AgentUpdate\PU\apu-12.60\"
If there is a "Deprecated" folder move this to a safe location. In the "Current" directory move any folder that is not the HF APU currently being used to a safe location. Then copy in the HF APU here.
Now on EACH CCS Manager navigate to the following location. "<InstallDir>\DPS\AgentUpdate\PU". If using the 12.5.2 HF if the apu-12.60 exists move it to safe location. If using the 12.6.0 HF move the contents of apu-12.60 to a safe location.
Restart all the CCS Managers in the Load Balancer role.
We are now ready to run the APU job(s), once run this will update the agents to either the 12.5.2 or 12.6.0 version (depending on the HF used) with the newer certificates.
At this point we can reverse the steps and restore the 12.6.1 APU files that we moved to safe location in previous steps. Once those are done, in the LiveUpdate workspace make sure to "Check Updates", once it shows the 12.6.1 APU Again, we are now ready to upgrade the patched agents to 12.6.1.
Run the APU job again to now upgrade the agents to the 12.6.1 version.

Additional Information

NOTE: If the steps above do not work, then try the following highlighted extra steps.

Backup the current 12.6.1 APU on the AppServer "C:\ProgramData\Symantec\CCS\LiveUpdateStaging\CCS_AGENT_PU"
Move the "CCS1260.12_60_10100_1500_CCS_AGENT_PU" to a known safe location.
Take the 12.6.0 HF zip file extract out the APU zip and extract out the APU OR 12.5.2 HF
Copy the "CCS1260.12_60_10003_1300_CCS_AGENT_PU" folder OR "CCS1250.12_50_10205_1790_CCS_AGENT_PU" to the "C:\ProgramData\Symantec\CCS\LiveUpdateStaging\CCS_AGENT_PU"
Now in the CCS Console navigate to the LiveUpdate workspace, from the "Common Tasks" drop down menu select "Check Updates". Wait until this job finishes and shows the new Agent APU version.
Now we need to finish clean up the 12.6.1 APU files. On the Application server navigate to "<InstallDir>\Application Server\AgentUpdate\PU\apu-12.60\"
If there is a "Deprecated" folder move this to a safe location. In the "Current" directory move any folder that is not the HF APU currently being used to a safe location. Then copy in the HF APU here.
Now on EACH CCS Manager navigate to the following location. "<InstallDir>\DPS\AgentUpdate\PU". If using the 12.5.2 HF if the apu-12.60 exists move it to safe location. If using the 12.6.0 HF move the contents of apu-12.60 to a safe location.
For 12.5.2 HF move the contents of the apu-12.50 to a safe location, if the folder does not exist then create it. Now copy over the "ccsagentupdate.exe" (file found in folder from step 4 or 7) file from the 12.5.2 HF APU folder to the apu-12.50 folder, execute the file to extract out the contents.
For 12.6.0 HF after having moved the contents of the apu-12.60 to a safe location, copy the "ccsagentupdate.exe" (file found in folder from step 4 or 7) from the 12.6.0 HF APU folder to the apu-12.60 folder, execute the file to extract out the contents.
Now we are ready to run the APU job(s), once run this will update the agents to either the 12.5.2 or 12.6.0 version (depending on the HF used) with the newer certificates.
At this point we can reverse the steps and restore the 12.6.1 APU files that we moved to safe location in previous steps. Once those are done, in the LiveUpdate workspace make sure to "Check Updates", once it shows the 12.6.1 APU Again, we are now ready to upgrade the patched agents to 12.6.1.
Now run the APU job again to now upgrade the agents to the 12.6.1 version.