When running a Web Agent Option Pack (WAOP) and when a user tries to reach the Service Provider (SP) side after Identity Provider (IDP) authentication, the browser gets error 500 and the Federation Service reports:
[05/01/2020][11:53:51][18675][3560912640][][FWSBase.java][createSessionCookie][Placing smsession in browser [CHECKPOINT = SSO_PLACESMSSESSIONTOBROWSER_REQ]]
[05/01/2020][11:53:51][18675][3560912640][][FWSBase.java][createSessionCookie][Transaction with ID: 6f3d5b0e-9f7101d1-bead3e79-f5cdfd77-44451s2-2ba failed. Reason: FWSB_SESSION_COOKIE_CREATION_ERROR]
[05/01/2020][11:53:51][18675][3560912640][][FWSBase.java][createSessionCookie][Exception occured during SESSION cookie creation. Exception: An invalid domain [.example.com] was specified for this cookie]
[05/01/2020][11:53:51][18675][3560912640][][FWSBase.java][createSessionCookie][Ending the request processing with the HTTP response code: 500]
More, 2 SMSESSION cookies get created with 2 different domains. The Web Agent Option Pack create one with the following domain:
"Domain=.host.domain.domain.example.com"
which doesn't make sense because it is a machine name, not a domain, but still the cookie will only be used to that machine, and the Web Agent creates another one with
"domain=.domain.example.com"
Are both issues related?
Web Agent Option Pack on Tomcat 9.0.36 on RedHat 7;
Web Agent 12.52SP1CR09 on IIS 10 on Windows 2016;
Policy Server 12.8SP3 on Windows 2016;
Both issues aren't related.
Think that Web Agent Option Pack and the Web Agent aren't sharing the same ACO parameters, and the embedded Agent in the Web Agent Option Pack has a very limited list of ACO available (1).
As such, the Web Agent Option Pack is not designed to run as a Backend Web Agent which is proxied by another Web Agent. In the setting, the Web Agent running on IIS 10 is an ARR proxy to the backend Web Agent Option Pack.
So said, to fix the issue "Exception: An invalid domain", configure the LegacyCookieProcessor feature on Tomcat Server as per recommendations (2).
Also, fix the cookie domain which hasn't the same domain by configuring ARR on IIS with the following rules:
<proxy enabled="true" reverseRewriteHostInResponseHeaders="false" />