All Spectrum Secure Domain Connectors stopped working around the same time
search cancel

All Spectrum Secure Domain Connectors stopped working around the same time

book

Article ID: 281128

calendar_today

Updated On:

Products

Spectrum Network Observability

Issue/Introduction

We noticed that all of our Secure Domain Connectors (SDC) are displaying Critical alarms. The associated events show lost contact at around the same time on March 25, 2024.

The following is seen in the $SPECROOT/SDM/sdmLog.log file on the SpectroSERVER/SDM system where XXX.XXX.XXX.XXX is the ip address of the SDC system:

Sun Mar 24 10:00:05 2024: WARNING: SdmEtpkiConnectEndpoint run() ssock_handshake error. IP=XXX.XXX.XXX.XXX:6844, Port=6844, Thread=-1597298944

When running a sniffer on the SDM or SDC system, the following is seen in Wireshark:

Additionally, when running the following on the SpectroSERVER/SDM system from the $SPECROOT/SDM/cert directory, the following is seen:

openssl x509 -purpose -enddate -in SDMCAv2.pem
Certificate purposes:
SSL client : Yes
SSL client CA : Yes (WARNING code=3)
SSL server : Yes
SSL server CA : Yes (WARNING code=3)
Netscape SSL server : Yes
Netscape SSL server CA : Yes (WARNING code=3)
S/MIME signing : Yes
S/MIME signing CA : Yes (WARNING code=3)
S/MIME encryption : Yes
S/MIME encryption CA : Yes (WARNING code=3)
CRL signing : Yes
CRL signing CA : Yes (WARNING code=3)
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : Yes (WARNING code=3)
Time Stamp signing : No
Time Stamp signing CA : Yes (WARNING code=3)
notAfter=Mar 24 05:18:05 2024 GMT

Environment

Version: Any
Component: Spectrum Secure Domain Manager/Connector

Cause

By default, the Secure Domain Manager (SDM) and Secure Domain Connector (SDC) have security enabled.

Since the certificate has expired, the connection between the SDM and SDC is failing.

Resolution

There are three options to resolve this issue:

1. Run SDM/SDC in "-nosecure". If security is not needed, running in "-nosecure" disables HTTPS security, which is enabled by default. If the -nosecure option is used before any -remoteconnect or -accept entries, HTTPS is disabled for all connections.

To use the "-nosecure" option, please do the following:

On the SDC systems:

  • Shutdown the SdmConnectorService on the SDC system
  • Make a backup of the <SdmConnector>/bin/sdc.config file
  • Edit the sdc.config file and add -nosecure as the first line in the file. The following is an example (where X.X.X.X will be your ip address(es) of the SDM system(s) in your environment):

# Place config entries here, see README for options
-nosecure
-accept X.X.X.X

  • Save the change
  • On Linux, restart the SdmConnector by running the following commands:
    • systemctl stop sdmconnector
    • systemctl status sdmconnector  (Verify the SdmConnector has stopped)
    • systemctl start sdmconnector
    • systemctl status sdmconnector  (Verify the SdmConnector has started)
  • On Windows, the SdmConnectorService runs as a Service.
    • Open the Windows Services
    • Find the SdmConnector service
    • Stop the SdmConnector service
    • Start the SdmConnector service

On the SpectroSERVER/SDM systems:

  • Make a backup of the $SPECROOT/SDM/sdm.config file
  • Edit the sdm.config file and add -nosecure as the first line in the file. The following is an example (where X.X.X.X will be your ip address(es) of the SDC system(s) in your environment):

-nosecure
-remoteconnect X.X.X.X

  • Save the change to the file
  • Re-import the Secure Domain Manager Configuration in OneClick
    • Log into OneClick as an ADMIN user
    • In the Explorer tab, find and select the Secure Domain Manager folder
    • Click on the Information tab
    • Find and expand the Configuration folder
    • Click on the Import button to import the changes made to the sdm.config file

2. Create a new cert if security is needed

  • Log into the SDM/SpectroSERVER system as the root user or su/sudo su to the root user
  • cd to the $SPECROOT/SDM/cert directory
  • Rename the following files:
    • SDMCAKeyv2.pem
    • SDMCAv2.pem
    • SDMCertv2.p12
  • Run the following command where <Country Code> is the two letter Country Code for your environment. This will create new SDMCAv2.pem and SDMCAKeyv2.pem files.

./CertGen -t ca -c <Country Code>

  • Run the following command where <Country Code> is the same two letter Country Code used above for your environment. This will create a new SDMCertv2.01.p12 file:

./CertGen -t cert -c <Country Code>

  • Rename SDMCertv2.01.p12 to SDMCertv2.p12
  • Run the following from the $SPECROOT/SDM/certs directory on the SpectroSERVER/SDM system to verify the date of the new cert. Check the value of the "notAfter" parameter. The new cert should be good for two years.

openssl x509 -purpose -enddate -in SDMCAv2.pem

  • Re-import the Secure Domain Manager Configuration in OneClick
    • Log into OneClick as an ADMIN user
    • In the Explorer tab, find and select the Secure Domain Manager folder
    • Click on the Information tab
    • Find and expand the Configuration folder
    • Click on the Import button to import the changes made to the sdm.config file
  • Copy the following three files to the <SDMConnector>/bin/cert directory on the SDC system(s):
    • SDMCAv2.pem
    • SDMCertv2.p12

  • On Linux, restart the SdmConnector by running the following commands:
    • systemctl stop sdmconnector
    • systemctl status sdmconnector  (Verify the SdmConnector has stopped)
    • systemctl start sdmconnector
    • systemctl status sdmconnector  (Verify the SdmConnector has started)

  • On Windows, the SdmConnectorService runs as a Service.
    • Open the Windows Services
    • Find the SdmConnector service
    • Stop the SdmConnector service
    • Start the SdmConnector service

3. Use the certs in the attached New_SDM_Certs.zip file

  • To implement these new certs on the SDM/SpectroSERVER system
    • Copy the New_SDM_Certs.zip file to the SDM/SpectroSERVER system
    • Log into the SDM/SpectroSERVER system as the user that owns the Spectrum installation
    • Unzip the New_SDM_Certs.zip file
    • Copy the following three files to the $SPECROOT/SDM/cert directory
      • SDMCAv2.pem
      • SDMCertv2.p12
      • SDMCAKeyv2.pem
      • Re-import the Secure Domain Manager Configuration in OneClick
        • Log into OneClick as an ADMIN user
        • In the Explorer tab, find and select the Secure Domain Manager folder
        • Click on the Information tab
        • Find and expand the Configuration folder
        • Click on the Import button to import the changes made to the sdm.config file

  • To implement these new certs on the SDConnector system:
    • Copy the New_SDM_Certs.zip file to the SDConnector system
    • Log into the SDConnector system as the root user or su/sudo su to the root user
    • Unzip the New_SDM_Certs.zip file
    • Copy the following two files to the /opt/CA/SDMConnector/bin/cert directory
      • SDMCAv2.pem
      • SDMCertv2.p12

    • On Linux, restart the SDConnector by running the following commands:
      • systemctl stop sdmconnector
      • systemctl status sdmconnector  (Verify the SdmConnector has stopped)
      • systemctl start sdmconnector
      • systemctl status sdmconnector  (Verify the SdmConnector has started)

    • On Windows, the SdmConnectorService runs as a Service.
      • Open the Windows Services
      • Find the SdmConnector service
      • Stop the SdmConnector service
      • Start the SdmConnector service

 

Additional Information

  • If you had previously used the "-nosecure" option to work-around the issue until you generated a new cert or implement the attached New_SDM_Certs.zip file, before restarting the SdmConnectorService, edit the <SdmConnector>/bin/sdc.config file and remove the "-nosecure" parameter.
    Edit the $SPECROOT/SDM/sdm.config file and remove the "-nosecure" parameter. Then import the Secure Domain Configuration as described in the "Import the SDManager Configuration File" section of the DX NetOps  Spectrum documentation.
    New certs are planned out of the box for DX NetOps 23.3.8.
  • If you are setting up a Fault Tolerant SDM with Primary and Secondary keep in mind that  both SDMs should have the same certificates:
    • Copy the following files from Primary SDM to the $SPECROOT/SDM/cert directory on Secondary SDM
      • SDMCAv2.pem
      • SDMCertv2.p12
    • After the certificates have been deployed, restart both the SpectroSERVER on the SDManager hosts and the SDConnector process on the SDC hosts.



Attachments

New_SDM_Certs.zip get_app
Powered by Wolken Software