Everything is working fine when configured with OOTB credential collector but once a custom login page is used, the federation does not complete and throws error.
All SiteMinder versions.
During SiteMinder Federation, browser redirects multiple times and the querystrings get encoded.
There is typical URL encoding but also SiteMinder URL Encoding that takes place.
For example, %3D gets SiteMinder URL Encoded to -%3D or $%3D (instead of %253D) depending on your ACO setting for "LegacyEncoding".
These then gets URL Encoded by the custom login page to be %2D%3D or %24%3D.
And after submitting user credentials there is an error reporting one of the following:
1. Unable to resolve agent name for host '###.###.###.###'. Existing with HTTP 500 server error '00-0001'
2. "Input byte array has wrong 4-byte ending unit"
3. "Last unit does not have enough valid bits"
This is because the custom login page is not handling the Querystring as how the OOTB credential collectors do.
In the following screenshot you can see "-" appearing in multiple locations due to SiteMinder URL Encoding performed by the web agent before redirect.
Following - (or $) should be converted accordingly at the custom login page
LegacyEncoding=NO-: to %3A-/ to %2F-= to %3D-% to %25-? to ?-+ to %2B-# to %23-& to &-- to - |
LegacyEncoding=YES$: to %3A$/ to %2F$= to %3D$% to %25$? to ?$+ to %2B$# to %23$& to &$$ to $ |
Customer should capture a HAR file for their use case using the OOTB credential collector and then see how the querystring URL encoding/decoding works and apply exactly the same to their custom login page to make it transparent.