Sensors Disconnecting After Upgrading to Server Version 7.8.1

Products

Carbon Black EDR (formerly Cb Response) Carbon Black EDR

Issue/Introduction

After upgrading to 7.8.1, sensors start to drop offline. /var/log/cb/nginx/access.logs show 400 errors on /sensor/checkin or /sensor/register calls.

Environment

Carbon Black EDR: 7.8.1 or higher

Cause

Older EDR instances generated a cb-client-ca certificate set without a CA: True extension

Resolution

Getting Sensors Connected Again

Set the ssl client check to optional by editing /etc/cb/nginx/conf.d/includes/cb.server.base_sensor. Update line 2 (ssl_verify_client) from optional to off.
```
    ssl_verify_client       off;
```

In /etc/cb/cb.conf, disable the client check and revocation check.

update:
EnforceClientCerts=False

add: 
EnforceClientCertRevocation=False

Restart services and verify sensors are coming online.

standalone
/usr/share/cb/cbservice cb-enterprise restart

cluster
/usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start

Generating a New Client Certificate Set

Once we have verified sensors are able to connect and show online, the next step is to generate a new set of client certs.

Stop the EDR services

Backup the current expired client and server certificates.

mkdir /etc/cb/certs/expired_certs && cp /etc/cb/certs/cb-client* /etc/cb/certs/expired_certs/ && cp /etc/cb/certs/cb-server.* /etc/cb/certs/expired_certs/

Create new client certificates.
1. If EDR is installed on RHEL/CentOS 7 please follow this step, else continue to step 2. This temporary modifies the release file to will work around a el8 check in the utility code.
```
cp /etc/redhat-release /etc/redhat-release.bkp && sed -i 's/7/8/' /etc/redhat-release
```
2. Set an environmental variable to remove the accidental regeneration safety check.
```
export FORCE_REGENERATE=client-ca
```
3. Regenerate the cb-client-ca certificate set.
```
/usr/share/cb/cbssl certs --regenerate client-ca
```
4. Revert the OS version and unset the environmental variable
```
mv /etc/redhat-release.bkp /etc/redhat-release && unset FORCE_REGENERATE
```
Start the EDR services.
Revoke all sensor group certificates to generate new group certs signed by the newly generated cb-client-ca. How to Revoke Sensor Group Certificates
Allow some time for sensors to collect the updated sensor group certificates. This will depend on how many endpoints are actually online.

Note: If you have a Carbon Black EDR cluster deployment, you must synchronize the regenerated certificate across the cluster. Run the following command:
/usr/share/cb/cbcluster sync-certs --cert client-ca

Revert the Client Checks
Revert the settings to allow client certificate checks to proceed. Any sensor that did not get the update group certificates signed by the new cb-client-ca will drop offline. These will either have to be re-installed or these checks will need to be removed until they show online again.

Set the ssl client check to optional by editing /etc/cb/nginx/conf.d/includes/cb.server.base_sensor. Update line 2 (ssl_verify_client) back to optional.
```
    ssl_verify_client       optional;
```

In /etc/cb/cb.conf, re-enable the client check and revocation check.

update:
EnforceClientCerts=True

add: 
EnforceClientCertRevocation=True

Restart services and verify sensors are coming back online. This verifies the new client ca and group certificates are working.

Additional Information

Release notes goes over using cbssl scan prior to upgrade. 7.8.1 Release Notes
7.8.1 uses a newer version of OpenSSL and Nginx that have stricter validation against certificates, where signing certs missing the extension CA: True are denied, causing the handshake to fail at the Nginx level.