Thread pool exhaustion with validation server when access-challenge is enabled causes validation attempts to fail.

Products

• VIP Service

Issue/Introduction

Per RADIUS RFC 2865, 5.24, an access-challenge request and response can be identified by a unique string identifier in the RADIUS communication between the client and the VIP Enterprise Gateway Validation (RADIUS) Server. VIP EG uses the STATE attribute as the identifier to establish the process thread the transaction will consume. The process thread is released when the transaction is complete.

Cause

If the VIP EG Validation Server receives a STATE attribute value that is invalid or empty, a thread is consumed but the VIP EG validation server may not relinquish the thread. If this happens multiple times, a domino effect could eventually cause the invalid requests to overwhelm and exhaust all available threads. Transactions begin to timeout and fail until the queue can automatically clear or the VIP EG validation service is restarted. This condition can occur when using Access-Challenge with the Palo Alto GlobalProtect Client. 

RADIUS RFC 2865, 5.24:

The STATE Attribute is available to be sent by the server to the client in an Access-Challenge and MUST be sent unmodified from the client to the server in the new Access-Request reply to that challenge, if any. This Attribute is available to be sent by the server to the client in an Access-Accept that also includes a Termination-Action Attribute with the value of RADIUS-Request. If the NAS performs the Termination-Action by sending a new Access-Request upon termination of the current session, it MUST include the State attribute unchanged in that Access-Request. In either usage, the client MUST NOT interpret the attribute locally. A packet must have only zero or one State Attribute. Usage of the State Attribute is implementation dependent.

This String field contains one or more octets.  The actual format of the information is site or application specific, and a robust implementation SHOULD support the field as undistinguished octets. The codification of the range of allowed usage of this field is outside the scope of this specification.

A Wireshark packet capture of the UDP traffic between the ports will show the STATE value in the RADIUS request/response. To decrypt the RADIUS traffic:

• Launch Wireshark, then capture the traffic.
• Filter by UDP protocol and the port.
• Click Edit > Preferences.
• Click the + next to Protocols to expand.
• Select RADIUS.
• Type in the RADIUS shared secret, then click Apply. (Note: Type the passcode in clear text.)
• The UDP traffic will be decrypted to the RADIUS traffic.

Resolution

Contact your VPN vendor and share the Wireshark capture and other logs with them. Often, a patch or update can be applied to address the invalid STATE being sent to the VIP Validation Server.

As a workaround, the patch file attached to this article can be applied to the VIP Enterprise Gateway server(s) 9.8.3 or later only. 

1. Download and extract the attached VSValidationServer.zip. 
2. Log in to the VIP Enterprise Gateway on the Windows server
3. Stop all validation servers
4. Navigate to the EG installation path (<install_Dir>\Validation\bin)
5. Rename the existing VSValidationServer.exe to VSValidationServer.exe.old
6. Copy the VSValidationServer.exe extracted from step 1 into this folder.
7. Start All validation servers

This step will force the validation servers on the VIP Enterprise Gateway to ignore access-challenge traffic containing invalid or null STATE attributes that do not comply with RADIUS RFC 2865. 

Note: The attached file is a hotfix patch. The permanent fix will be built into an upcoming release of VIP EG.