The checkbox for "Allow Nested Groups" is not present when you are creating a SAML Service Provider object under Legacy Federation, or if you are modifying a SAML Service Provider object. When you finish creating the object or saving an existing object, it will not have the "Allow Nested Groups" option enabled if it was not enabled prior, and it will be disabled if it was enabled before.
Applies to all environments for the specified releases with non-LDAP User Directories
The code for the AdminUI has an explicit check that only displays the "Allow Nested Groups" checkbox if the User Directory is an LDAP directory.
If you made changes to the Affiliate Domains and need to enable the "Allow Nested Groups" option, please follow the instructions below:
1. Run XPSExplorer
2. Go to the AffiliateDomain (should be 94, but confirm)
3. Display Related Records (R)
4. Select CA.SM::UserPolicy.DomainLink (1)
5. Select the appropriate UserPolicy
6. Get a writeable copy by entering W
7. Select PolicyFlags (should be 4)
8. Enable "Allow Nested Groups" by entering the value "2"
9. Validate the update by entering V
10. Update by entering U.
11. Quit out of XPSExplorer
This was resolved in Defect 157370, and will be implemented in 12.52 SP1 CR05, and 12.52 SP2 CR01.