This is a known issue in SGOS 6.2.x (B#197914) and in SGOS 6.4.x (B#190218).

The issue occurs when the server extension is not recognized by the server, and the server returns a warning message (Unrecognized name). Previously, the appliance would drop the connection per the recommendation of SSL implementation. With the existence of the TLS (renegotiation) the behavior has changed to allow the client to re-negotiate in case of a warning response. 

Example of PCAP file:
 

--- client sent Client Hello
4    2014-01-27 17:55:57.333999    10.169.0.53        12.173.81.44    TLSv1    241        Client Hello
<--- SG sent the same client hello to the OCS
8    2014-01-27 17:55:57.363000    10.169.3.178    12.173.81.44    TLSv1    241        Client Hello
<--- OCS sent Server Hello with Alert(Unrecognized Name) as well as certificate
9    2014-01-27 17:55:57.392999    12.173.81.44    10.169.3.178    TLSv1    1434        Alert (Level: Warning, Description: Unrecognized Name), Server Hello
10    2014-01-27 17:55:57.392999    12.173.81.44    10.169.3.178    TLSv1    150        Certificate
<--- SG acknowledged the server hello & certificate, and sent the Alert to the client.
11    2014-01-27 17:55:57.392999    10.169.3.178    12.173.81.44    TCP    60        59232 > 443 [ACK] Seq=62545189 Ack=3036911236 Win=65164 Len=0
12    2014-01-27 17:55:57.392999    12.173.81.44    10.169.0.53     TLSv1    61        Alert (Level: Warning, Description: Unrecognized Name)
<--- SG then closed both server and client connections.
13    2014-01-27 17:55:57.394000    10.169.3.178    12.173.81.44    TCP    60        59232 > 443 [FIN, ACK] Seq=62545189 Ack=3036911236 Win=65535 Len=0
14    2014-01-27 17:55:57.394000    12.173.81.44    10.169.0.53        TCP    60        443 > 53019 [FIN, ACK] Seq=2078261425 Ack=2858178668 Win=65535 Len=0

For a ProxySG 510 appliance, Blue Coat recommends upgrading to SGOS 6.4.6.3.

For other appliances, upgrade to SGOS 6.2.16.1 or SGOS 6.5.5.1.