One of customer's federation link throws a blank page on the first attempt with http error 400, subsequent attempts may work fine but not consistently.
No other obvious errors were observed anywhere and this was a working setup.
After examining Tomcat server log, where agent option pack was deployed, saw the following errors:
24-Apr-2019 22:35:58.004 SEVERE [ajp-bio-8009-exec-1] org.apache.coyote.ajp.AjpMessage.processHeader Invalid message received with signature 514 ontainerBase$StopChild.call(ContainerBase.java:1579) ...
24-Apr-2019 21:45:22.493 SEVERE [localhost-startStop-1] org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: org.apache.log4j.Logger at java.io.ObjectInputStream.readObject0(Unknown Source)
The particular federation request may have gone through url redirection.
And during url redirection, the size of http packet may change, and sometimes it could surpass browser, AJP connector or application server buffer limit.
In this case, packetSize attribute for AJP connector is culprit.
SSO Policy Server: 12.52 SP1 CR 5
SSO Policy Server OS: WIN 2008 R2
Web Agent: 12.52 SP1
Web Server: IIS 7
Web Server OS: WIN 2008 R2
Customer resolved it by adding the packetSize attribute for Tomcat AJP connector.
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" packetSize="65536" />