The REST API for Authenticating and Authorizing users is part of the SPS (Access Gateway) Server called the Authentication and Authorization Web Services.
The documentation can be found here:
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/configuring/ca-access-gateway-configuration/configuring-the-authentication-and-authorization-web-services.html
The Section about the REST API are here:
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/configuring/ca-access-gateway-configuration/configuring-the-authentication-and-authorization-web-services.html#concept.dita_83589c7cb4c6518eba67ea69ba998c95c546b285_AuthenticationRESTInterface