ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Investigating and resolving the "Schannel (xxx_IWA_DIRECT): Resetting Schannel due to error: 0xC0000001(-1073741823)" error

book

Article ID: 237344

calendar_today

Updated On:

Products

ProxySG Software - SGOS

Issue/Introduction

SWG Proxy IWA Authentication Service keep failing- Internet Down 

 

2022-03-19 07:03:37-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-19 07:03:47-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384
2022-03-19 12:51:34-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-19 12:51:44-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:12:30-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:12:50-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: Check failed, and has become sick. Status: Status successful."  0 3A0003:1  hc_container.cpp:382
2022-03-20 00:17:52-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: Check failed, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:18:12-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:18:32-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: Check failed, and has become sick. Status: Status successful."  0 3A0003:1  hc_container.cpp:382
2022-03-20 00:25:45-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: Check failed, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:25:55-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:26:15-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: Check failed, and has become sick. Status: Status successful."  0 3A0003:1  hc_container.cpp:382
2022-03-20 00:28:36-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: Check failed, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:28:56-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK, to: OK but failing, and has become useable. Status: Status successful."  0 3A0003:96  hc_container.cpp:384
2022-03-20 00:29:16-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: OK but failing, to: Check failed, and has become sick. Status: Status successful."  0 3A0003:1  hc_container.cpp:382
2022-03-20 00:29:36-04:00EDT  "Health check 'auth.corp_iwa_direct' changed from: Check failed, to: OK, and has become useable. Status: Success."  0 3A0003:96  hc_container.cpp:384

2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225473 (0xC0000001) (symbol: '<null>'): user 'CNandiga' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225473 (0xC0000001) (symbol: '<null>'): user 'CDraper' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Connected to DC: sxpcdc03.corp.dtcc.com"  0 250041:96  lw_schannel.cpp:1338
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Resetting Schannel due to error: 0xC0000001(-1073741823), DC: sxpcdc03.corp.dtcc.com"  0 250042:1  lw_schannel.cpp:827
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225473 (0xC0000001) (symbol: '<null>'): user 'JJuliano' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Connected to DC: sxpcdc03.corp.dtcc.com"  0 250041:96  lw_schannel.cpp:1338
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Resetting Schannel due to error: 0xC0000001(-1073741823), DC: sxpcdc03.corp.dtcc.com"  0 250042:1  lw_schannel.cpp:827
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225997 (0xC000020D) (symbol: '<null>'): user 'GOliver' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225473 (0xC0000001) (symbol: '<null>'): user 'CRicero' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Refreshing machine TGT for domain CORP.DTCC.COM"  0 250054:96  sg_syslog.cpp:304
2022-03-20 01:19:35-04:00EDT  "Finished refreshing machine TGT for domain CORP.DTCC.COM, result 0x0(0)"  0 250054:96  sg_syslog.cpp:304
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Connected to DC: sxpcdc03.corp.dtcc.com"  0 250041:96  lw_schannel.cpp:1338
2022-03-20 01:19:35-04:00EDT  "Schannel (CORP_IWA_DIRECT): Resetting Schannel due to error: 0xC0000001(-1073741823), DC: sxpcdc03.corp.dtcc.com"  0 250042:1  lw_schannel.cpp:827
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225997 (0xC000020D) (symbol: '<null>'): user 'GOliver' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Authentication failed with 3221225473 (0xC0000001) (symbol: '<null>'): user 'SGudipat' (domain CORP) - user considered 'unknown'"  0 250017:96  lw_schannel.cpp:593
2022-03-20 01:19:35-04:00EDT  "Refreshing machine TGT for domain CORP.DTCC.COM"  0 250054:96  sg_syslog.cpp:304
2022-03-20 01:19:35-04:00EDT  "Finished refreshing machine TGT for domain CORP.DTCC.COM, result 0x0(0)"  0 250054:96  sg_syslog.cpp:304

Resolution

For the authentication failed state, in Health Checks on the ProxySG, investigating the IWA Server "Test Configuration" error received, as seen in the PCAP snippet below, it was confirmed that, for the communication between the ProxySG and the authentication server (DC), there was, indeed, no data in the channel.

For the response received in frame 136, Microsoft's description of the error is seen in the Microsoft doc. with the URL below.

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/320f04f3-1b28-45cd-aaa1-9e5aed810dca

Note 1:

Netlogon is a Windows Server process that authenticates users and other services within a domain. Since it is a service and not an application, Netlogon continuously runs in the background, unless it is stopped manually or by a runtime error.

Netlogon found no data in the channel. The "Test Configuration" task failed with the "The provider credentials did not match" error.

Note 2:

When "Test Configuration" is triggered, the appliance sends an authentication request to the configured server and then displays a message indicating whether the authentication succeeded or failed. If the test failed, go back and make sure you have configured the realm properly. If the test succeeds, the message also displays a list of any groups of interest (that is, groups that are referenced in the policy) to which the user belongs. Where the realm configuration is confirmed to be correct, the other reason would be because Netlogon could not create a secure channel with the client (The ProxySG). Netlogon service failed.

Note 3: Netlogon service on Domain Controller is responsible for creating Secure Channel between Domain Controllers and clients. Secure Channel is created to pass the authentication packets. 

To further validate this, we looked into the uploaded eventlog and we found that DC: sxpcdc03.corp.dtcc.com, very intermittently, kept resetting the secured channel. The Schannel reset happened 6870 times, as seen in the logs.  Excerpts of the resets are shown in the introduction.

Causes:

  • Netlogon Schannel reset.
  • DC Unreachable
  • Possibly, the local firewall was blocking the ports that are required for authentication to function

Resolution:

A rejoin of the ProxySG appliance to the Windows domain was attempted again, and this time, the Schannel (CORP_IWA_DIRECT): Connected to DC: sxpcdc03.corp.dtcc.com and the "authentication" states, in Health Check" returned to the "OK" state.  In this state, we also saw that the "Test Configuration", in IWA Servers, happened, successfully. See the snippet below.

Recommendation

Because this is, clearly, an intermittent behavior with NETLOGON reset of the Schannel, it's strongly recommended that you diligently implement the recommended Proxy-side and DC-side resolution, and all other guidances, before putting the affected ProxySG appliance back in production, and to prevent future occurrence. 

Ref. docs.:

https://knowledge.broadcom.com/external/article/175348/troubleshooting-steps-for-failed-authent.html

https://knowledge.broadcom.com/external/article/166031/implement-integrated-windows-authenticat.html

Attachments