In IDP to SP Federation Partnership there is "IP Restrictions" where "Allowed Addresses and Hosts" can be set.
Only the users authenticated with those IP address should be allowed to federate but this IP restriction was not being enforced.
R12.8SP8 Access Gateway and Policy Server
ACO must have IP Checking enabled.
As IP Checking was not enabled, IP Restriction does not get enforced.
All the Agents that are handling the Federation requests must have respective IP Checking enabled.
If Transient Cookie is used, then TransientIPChecking must be enabled(set to "yes").
PersistentIPChecking is enabled by default.