Regenerating a New Set of Certificates for Sensor to Server Communication when the Originals Expire
search cancel

Regenerating a New Set of Certificates for Sensor to Server Communication when the Originals Expire

book

Article ID: 376576

calendar_today

Updated On:

Products

Carbon Black EDR (formerly Cb Response)

Issue/Introduction

When the original sensor to server certificates expire, the sensors will no longer communicate with the server. This article goes over getting the sensors connected back up, generating a new set of client and server (self-signed) certificates that can be swapped out for continued communication.

Environment

  • Carbon Black EDR Server: 7.7.0 and higher

Cause

The default certificates are good for 10 years. 

Resolution

 Getting Sensors Connected Again

  1. Set the ssl client check to optional by editing /etc/cb/nginx/conf.d/includes/cb.server.base_sensor. Update line 2 (ssl_verify_client) from optional to off. 
        ssl_verify_client       off;
  2. In /etc/cb/cb.conf, disable the client check and revocation check. 
    update:
EnforceClientCerts=False

add: 
EnforceClientCertRevocation=False
  3. Restart services and verify sensors are coming online. 
    standalone
/usr/share/cb/cbservice cb-enterprise restart

cluster
/usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

 

Generating a New Client Certificate Set

Once we have verified sensors are able to connect and show online, the next step is to generate a new set of client certs. 

  1. Stop the EDR services
  2. Backup the current expired client and server certificates. 
    mkdir /etc/cb/certs/expired_certs && cp /etc/cb/certs/cb-client* /etc/cb/certs/expired_certs/ && cp /etc/cb/certs/cb-server.* /etc/cb/certs/expired_certs/
  3.  Create new client certificates.
    1. If EDR is installed on RHEL/CentOS 7 please follow this step, else continue to step 2. This temporary modifies the release file to will work around a el8 check in the utility code. 
      cp /etc/redhat-release /etc/redhat-release.bkp && sed -i 's/7/8/' /etc/redhat-release
    2. Set an environmental variable to remove the accidental regeneration safety check. 
      export FORCE_REGENERATE=client-ca
    3. Regenerate the cb-client-ca certificate set. 
      /usr/share/cb/cbssl certs --regenerate client-ca
    4. Revert the OS version and unset the environmental variable 
      mv /etc/redhat-release.bkp /etc/redhat-release && unset FORCE_REGENERATE
  4. Start the EDR services.
  5. Revoke all sensor group certificates to generate new group certs signed by the newly generated cb-client-ca. How to Revoke Sensor Group Certificates 
  6. Allow some time for sensors to collect the updated sensor group certificates. This will depend on how many endpoints are actually online.

 

Revert the Client Checks

Revert the settings to allow client certificate checks to proceed. Any sensor that did not get the update group certificates signed by the new cb-client-ca will drop offline. These will either have to be re-installed or these checks will need to be removed until they show online again. 

  1. Set the ssl client check to optional by editing /etc/cb/nginx/conf.d/includes/cb.server.base_sensor. Update line 2 (ssl_verify_client) back to optional. 
        ssl_verify_client       optional;
  2. In /etc/cb/cb.conf, re-enable the client check and revocation check. 
    update:
EnforceClientCerts=True

add: 
EnforceClientCertRevocation=True
  3. Restart services and verify sensors are coming back online. This verifies the new client ca and group certificates are working. 

 

Generating a Temporary Self-signed Certificate 

The next step is to generate a temporary server certificate once all sensors are online with the checks still enforced. (Sensors will drop offline in the console after 5 minutes if the procedure failed).

  1. Copy the script from the additional information section and create a temp_server_cert.sh file to the EDR server backend.
    • It's recommended to create a new directory so the cert files are easy to find. cd into this directory as it will drop the files into the current working directory.  
  2. Update the script permissions 
    chmod 700 temp_server_cert.sh
  3. View the script, you can change any of the parameters under the EOF section that will be entered into the script or leave it default. The DNS.1 and DNS.2 entries can be left default or modified, however they should not be a DNS resolvable name. 
  4. Run the script with a "filename" indicator, hostname is suggested. 
    ./temp_server_cert.sh $(hostname)
  5. This will drop three files, the .conf to make the cert and the .key and .pem. 
  6. Log into the EDR console as a global admin.
  7. Go to your username > Settings > Server Certificates
  8. Click to + Add certificate
    1. Add a name, this is alphanumeric restricted, no spaces or special characters. "temporary" can be used to help distinguish for deletion later.
    2. Upload the .pem under the "Upload certificate" section
    3. Upload the .key under the "Upload private key" section
    4. Click Add to create the temp cert.
  9. Go to the Sensors page
  10. Find a group with minimal sensors that are easily accessible incase they need to be reinstalled. Or create a new group and assign a sensor into it. 
  11. Edit the group settings by setting the "Assign Server Certificate" to the newly created temp certificate.
  12. Wait 10 minutes to verify the sensor still shows online. 
  13. If the sensor still shows online, edit each sensor group with this newly created temp certificate.
  14. Wait until all active sensors show this new "temporary" certificate name in the sensors page "Server Certificate" column before going to the next steps. 

Generate a New "Legacy" Self-signed Certificate

  1. Stop the EDR services.
  2. Set a new environmental variable. 
    export FORCE_REGENERATE=legacy
  3. If EDR is installed on RHEL/CentOS 7 please follow this step, else continue to step 2. This temporary modifies the release file to will work around a el8 check in the utility code.
    cp /etc/redhat-release /etc/redhat-release.bkp && sed -i 's/7/8/' /etc/redhat-release
  4. Create the new legacy self signed certificate. 
    /usr/share/cb/cbssl certs --regenerate legacy
  5. Revert the OS version file change and unset the protection environmental variable. 
    mv /etc/redhat-release.bkp /etc/redhat-release && unset FORCE_REGENERATE
  6. Start the EDR services.
  7. Log back into the console.
    1. Go into the username > Settings > Server Certificates
    2. Verify the "legacy" certificate has a new thumbprint and non-expired date.
    3. Go to the sensors page and edit the test group by setting "legacy" as the "Assign Server Certificate"
    4. Wait 10 minutes to verify the sensor stays online. 
  8. If the sensor stays online, modify each sensor group with this new legacy certificate.
  9. When all sensors are now using the new legacy server certificate, the temporary certificate can be deleted from the console. 
  10. If on an EDR Cluster run this command to sync the certs on the Primary Server to the Secondary Servers 
    /usr/share/cb/cbcluster sync-certs

Additional Information

  • Due to the nature of certificates and their purpose, the steps are complicated to swap out the certificates. This option provides the ability to swap certificates with minimal re-installation required. There may still be sensors that need installation after following these instructions if the endpoint was not online during the time the steps were performed. 
  • Any sensor installer package created prior to this change will need to be regenerated. Failure to do this will result in new sensors not connecting. Verify with your teams that use an application management software such as JAMF, SCCM, GPO etc. that they are using newly generated packages for the installation automation.
  • The other option is to re-install all sensors with a newly generated package. If this is the option you prefer, follow the steps outlined in these two sections only prior to downloading sensor installation packages. 
    • Generating a New Client Certificate Set
    • Generate a New "Legacy" Self-signed Certificate

temp_server_cert.sh

#!/bin/sh

: '

### THIS SCRIPT GENERATES A TEMPORARY SELF SIGNED CERT WITH TWO SAN ENTRIES ###

When running the script, please add a domain name. 
This is just used to name the files, however it may be easier to use the hostname.

./temp_server_cert.sh <domain name> 

The two DNS entries are the SAN entries required for a custom certificate. 
DNS.1
DNS.2
These can be anything you want as long as they are not DNS resolvable. Or you can use the default entered already.

Feel free to update certificate entry information

'


if [ "$#" -ne 1 ]
then
  echo "Usage: Must supply a domain"
  exit 1
fi
 
DOMAIN=$1
 
cat << EOF >> ${DOMAIN}_config.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = MA
L = Waltham
O = Broadcom Carbon Black
OU = Support
CN = XXXX
[email protected]
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = edr.primary
DNS.2 = edr.minion
EOF
 
echo "Creating the self signed cert and key"
openssl req -x509 -nodes -days 1825 -newkey rsa:2048 -keyout ${DOMAIN}_private_sensor.key -out ${DOMAIN}_public_sensor.pem -config ${DOMAIN}_config.conf -extensions 'v3_req'
Powered by Wolken Software