Using SHA-256 with RSA certificates for Data Center Security server and agent communication

book

Article ID: 169435

calendar_today

Updated On:

Products

Data Center Security Server Advanced

Issue/Introduction

In Data Center Security: Server Advanced, we generate self-signed certificates while installing the Management Server. These self-signed certificates are required by the agent installer, and subsequently used by the IPS service (OpenSSL module) to authenticate the server as part of the SSL communication. The out of the box certificates generated in 6.7 MP1 and earlier use SHA-1 as the default signature algorithm and key length of 2048.  In 6.7 MP2 and later, new certificates are generated using SHA 256 with RSA by default and the following steps are not required unless you are upgrading an existing infrastructure.

Note: Older frozen platforms may still need SHA-1 certificates for communication between the agent and the management server.  

Environment

Data Center Security: Server Advanced 6.7 MP1 and earlier.

Resolution

To generate new certificates using SHA-2

Perform the following tasks on Data Center Security Management Server to generate new certificates using SHA-2 family:

Note: In this article, SHA-256 hash algorithm of SHA-2 family is used.

Preparing for manual generation of SHA-256 certificates

  1. Back up the original certificate files to a safe location.
    The certificates can be found at the following locations:
  • <DCS server Install Directory>\server\agent-cert.ssl
  • <DCS server Install Directory>\server\server-cert.ssl
  1. Ensure that you have access to the server.xml filelocated at:

<DCS server Install Directory>\server\tomcat\conf

From the server.xml file, record the value for keystorepass, which is an alphanumeric string of 40 characters.

  1. Record the Common Name (CN) parameter. For the Data Center Security: Server Advanced server, this value is always SCSP_Management_Server.
  2. Record the Hostname of the Management Server, which you need for the OU parameter.
  3. Locate the keytool.exe that is present at the following location:

<DCS server Install Directory>\server\jre\bin

Creating the SHA-256 certificates manually

  1. From the command-line, access the keytool utility that is present at the following location:

<DCS server Install Directory>\server\jre\bin

  1. Create a temporary folder, for example: C:\TempDCS\
  2. Copy the server-cert.ssl to this temporary location C:\TempDCS\
  3. Using the command line, enter the following:

keytool.exe -delete -keystore C:\TempDCS\server-cert.ssl -alias sss -storepass [40 character alpha-numeric string that is found in the server.xml file] -storetype PKCS12

  1. Using the command line, enter the following:

keytool.exe -genkey -keystore “C:\TempDCS\server-cert.ssl” -alias sss -keyalg RSA -sigalg SHA256withRSA -keysize 2048 -storetype PKCS12 -storepass [40 character alpha-numeric string found in the server.xml file] -keypass [40 character alpha-numeric string found in server.xml] -dname "CN=SCSP_Management_Server, OU=[SCSP server hostname]"

  1. Using the command line, enter the following:

keytool.exe -export -Alias sss -rfc -keystore “C:\TempDCS\server-cert.ssl” -file “C:\TempDCS\agent-cert.ssl” -storepass [40 character alpha-numeric string found in the server.xml file] -storetype PKCS12

  1. Use the agent-cert.ssl created in Step 6 for agent-server communication.

Replacing existing certificates with newly created certificates

  1. On the Data Center Security: Server Advanced server, do the following:
  • Stop the Data Center Security: Server Advanced management service
  • Replace the original server-cert.ssl found at the following location with the new certificates created by using keytool:
    <DCS server Install Directory>\server 
  • Replace the original server-console-cert.ssl found at the following location with the new certificates created by using keytool:
    <DCS server Install Directory>\server
  • Replace the original agent-cert.ssl that is present at the following location with the new agent-cert.ssl created by using keytool:
    <DCS server Install Directory>\server
  • Start the Data Center Security: Server Advanced management server service
  • If you are using Data Center Security: Server, restart the SVA virtual machines in your VMWare platforms.
  1. On the Data Center Security: Server Advanced agent, do the following:
  • Copy the newly created agent-cert.ssl to the agent machine.
  • Update the agent to use the new agent-cert.ssl with the following command (forces use of new agent-cert.ssl file):
    sisipsconfig -c agent-cert.ssl
  • To test the connection from the command prompt:
    sisipsconfig -t

Limitation

Limitation of using SHA-256 certificate:
If Management Server and agent certificates are created using SHA-256 algorithm, then the legacy agents that are mentioned in the following table may fail to communicate with the Management Server.

Workaround: 
You must install tomcat only server and use certificates using SHA-1 for agent communication, and register legacy agents with this tomcat only server.

Note: If you are using the following agent versions, then you must use SHA-1 certificate:

Frozen agent binary from 5.2.9.MP6

Version

agent64-linux-rhel4.bin

5.2.7-184

agent-aix5L_51_52.bin

5.2.0-510

agent-esx3.bin

5.2.5-159

agent-hpux11i_v1_v2-hppa.bin

5.2.0-510

agent-hpux11i_v1_v2-ia64.bin

5.2.0-510

agent-linux-rhel3.bin

5.2.0-563

agent-linux-rhel4.bin

5.2.7-184

agent-linux-rhel4-ia64.bin

5.2.7-184

agent-linux-sles8.bin

5.2.0-519

agent-linux-sles9.bin

5.2.0-519

agent-solaris8-sparc.bin

5.2.0-519

agent-solaris9-sparc.bin

5.2.8-337

agent-tru64.bin

5.2.0-510