EDR: Windows Sensordiags Report HrError 0x80072F76 when Submitting Data

Products

• Carbon Black EDR (formerly Cb Response)

Issue/Introduction

• Sensordiag reports HrError[0x80072F76] for all submits while reserves are successful.
• CBdiag nginx access.logs report HTTP 200 for the sensor reserves but no submit attempts were recorded.

Environment

• EDR Sensor: Windows 7.3.0, 7.3.1

Cause

The API that is failing is WinHttpAddRequestHeaders which is a preparatory step to the submit transmission. This API is getting a return code of 2F76 due to a malformed IP address after checking each IP addresses on the sensor.  While recently updating the code, the default IP address was removed.  

Resolution

This is sensor issue CB-39155. The fix is expected in Windows sensor version 7.3.2.  Please remain on version 7.2.2 if the sensor is encountering this issue.

A) To identify an impacted sensor from a sensordiag, look for the following errors in the Sensor.LOG file:

(w): WinHttpAddRequestHeaders() failed: WinError[0x00002F76: Unknown error 0x2F76]
(e): PutRawEventLog failed HTTPCode[2147954550] HrError[0x80072F76]
(w): Unable to push event log 'eventlog_12218468.10486.log.zip' HrError[0x80072F76]
(w): Unable to properly synch with server HrError[0x80072F76]

B) To identify impacted sensors from the EDR server, or cluster:
      1.  On each EDR server, gather a list of Sensors that request a 'reserve':

# grep reserve /var/log/cb/nginx/access.log | cut -d '/' -f 7 | sort -u > sensors-reserve.txt

       2.  Gather a list of Sensors that request a 'submit':

# grep submit /var/log/cb/nginx/access.log | cut -d '/' -f 7 | sort -u > sensors-submit.txt

       3.  Run 'sdiff' on the 2 resulting files:

# sdiff sensors-reserve.txt sensors-submit.txt > sdiff-results.txt

       4.  Get your final results:

# grep "<" sdiff-results.txt > final-output.txt

       5.  Advanced: If many sensors are affected move the them to a new group in bulk and downgrade to win-7.2.2.

In the EDR console, create a new group.
# cut -d ' ' -f 1 final-output.txt > sensorlist.txt
Make a comma-delimited list from sensorlist.txt:
# vi sensorlist.txt (Run ":%s/$\n/,/" to create a comma-delimited list.  Then save with ":wq" )
On the Primary EDR server, obtain the new group's ID.
psql cb -p 5002 -c "select id,name from sensor_groups;"
Move the list of sensors to the new group.
psql cb -p 5002 -c "update sensor_registrations set group_id=<new group id> where id in (<cut-and-paste comma-delimited sensor list>;"

Additional Information

• Some sensors with multiple IP addresses may receive this error, while other sensors with multiple addresses work properly.   
• There may be other, more efficient ways to identify impacted sensors than what is described above, but those steps should work.
• Gather sensor logs or use Live Response to manually confirm the presence of the 0x80072F76 error code.