How to troubleshoot and fix Operations Manager authentication issues with SAML IdP

Products

• Operations Manager

Issue/Introduction

This document contains steps on how to troubleshoot and fix Operations Manager (Ops Manager) authentication issues with IdP SAML.

Ops Manager (v2.x) authentication has been switched from Internal to SAML IdP. The account which is supposed to have admin privileges is not getting authorized as such by Ops Manager.

It seems like the authentication is working but the authorization is not working properly, as hinted at by the following message in the UI:

This is likely caused by the Role Based Access Control (RBAC) fields ("SAML Admin Group" and "Groups Attribute") values not matching with what is set up in the IdP.

Environment

Product Version: 2.0

Resolution

To resolve this issue, follow the steps below:

1. Using your Chrome browser, download and install the "SAML Chrome Panel". This will be used to check the SAML response message.

Note: You can also use Firefox, or any browser, with an extension that can monitor and present SAML messages.



2. Go to the Ops Manager UI and log out by using the URL: https://OPSMAN.DOMAIN/logout

3. Go to the Ops Manager login page at: https://OPSMAN.DOMAIN

Note: Upon the page loading, you should then be redirected to the IdP page wherein you'd enter your credentials but hold off on doing anything for now. 

4. At the IdP page, invoke the Developer Tools by clicking from the menu View, select Developer, and select Developer Tools.
 

5. Once the Developer tools are invoked, click on the >> found on the right side of the menu, and click SAML.



6. In the IdP page, proceed to enter your credentials and log in. In the SAML tool, you should see a SAML response message similar to the one below, with the path pointing to your Ops Manager UAA URL. Analyze the SAML response message and find the Attribute and Attribute Value that likely corresponds to the Admin group name and group attribute.

Note: These values have no uniform format across various IdP's. Therefore, a confirmation from the IdP team (of the customer) might be helpful if you cannot figure them out. In the below example, the Attribute Name is external_groups and the Attribute Value is pcfadmins.

7. Perform this step to know what is actually configured in Ops Manager. Log on to Ops Manager VM and execute the following command to know what is configured in Ops Manager.

$ echo "select rbac_saml_admin_group,rbac_saml_groups_attribute from uaa_configs" | sudo -u tempest-web psql tempest_production
 rbac_saml_admin_group | rbac_saml_groups_attribute
-----------------------+----------------------------
 pcfadmins             | memberof
(1 row)

$

8. Analyze and compare the values between the IdP and Ops Manager. The value of IdP's Attribute Name must match the value of Ops Manager's rbac_saml_admin_attribute. The value of IdP's Attribute Value must match the value of Ops Manager's rbac_saml_admin_group.

As you would notice in the above example, the IdP's Attribute Name external_groups doesn't match Ops Manager's rbac_saml_admin_attribute "memberof", hence the error message.

9. To fix the issue, you will have to analyze and decide where it makes sense (IdP or Ops Manager) to adjust so that the values would match. 

If you decide that it makes more sense to adjust it in the IdP, then have the value(s) corrected in the IdP and afterwards try to login into Ops Manager UI. Otherwise, if you decide to correct it in Ops Manager then first try to How to put Ops Manager into Rescue Mode. Once in rescue mode, you can switch authentication to Internal first, and then back to SAML IdP with the correct values.

If Ops Manager rescue mode is not working as with some versions then follow the procedures below in order to manually update the Ops Man databases to correct the values.

How to manually update the Ops Manager databases to correct the RBAC SAML values?

IMPORTANT STEP:  To manually correct the values in the Ops Man and UAA databases, log in to the Ops Man VM and back up the databases first in a persistent directory of your choice. 

Do not proceed before completing these backup steps.

$ sudo -u tempest-web pg_dump tempest_production -f tempest_production.sql
$ sudo -u tempest-web pg_dump uaa -f uaa.sql

 

If the value of the SAML Admin Group needs to be corrected in Ops Manager, then follow these steps.

 

a. Log in to the Ops Manager VM, then log in to the tempest_production database using PSQL client.

$ sudo -u tempest-web psql tempest_production

b. Confirm what is the current SAML Admin Group that is in the 'uaa_configs' table. In this example, the wrong value is 'pcfops' and the correct value is 'pcfadmins'.

tempest_production=# select rbac_saml_admin_group from uaa_configs;
 rbac_saml_admin_group
-----------------------
 pcfops
(1 row)

tempest_production=#

c. Update the table with the correct value.

tempest_production=# UPDATE uaa_configs SET rbac_saml_admin_group = 'pcfadmins' WHERE rbac_saml_admin_group = 'pcfops';
UPDATE 1
tempest_production=#

d. Confirm that the table now has the correct value.

tempest_production=# select rbac_saml_admin_group from uaa_configs;
 rbac_saml_admin_group
-----------------------
 pcfadmins
(1 row)

tempest_production=#

e. Exit out of the psql client by typing '\q' and pressing Enter.

tempest_production-# \q
$

f. Log in to the Ops Manager UAA database using PSQL client.

$ sudo -u tempest-web psql uaa

g. Confirm what is the current SAML Admin Group that is saved in 'external_group_mapping' table. Also note the group_id values as you will need to use these in the update statements in the next step.

uaa=# select * from external_group_mapping where external_group = 'pcfops';
               group_id               | external_group |          added          |         origin         | identity_zone_id
--------------------------------------+----------------+-------------------------+------------------------+------------------
 3374447f-d7ec-40ec-878f-87d2bf261ad3 | pcfops         | 2018-08-29 02:34:59.361 | external-saml-provider | uaa
 8baf7580-41fa-4e9c-a36e-2b138368388f | pcfops         | 2018-08-29 02:34:59.374 | external-saml-provider | uaa
(2 rows)

uaa=#

h. Update the table with the correct value. Replace the group_id values in the conditions, with the ones you gather from Step 7.

uaa=# UPDATE external_group_mapping SET external_group = 'pcfadmins' WHERE group_id = '3374447f-d7ec-40ec-878f-87d2bf261ad3' and external_group = 'pcfops';
UPDATE 1
uaa=# UPDATE external_group_mapping SET external_group = 'pcfadmins' WHERE group_id = '8baf7580-41fa-4e9c-a36e-2b138368388f' and external_group = 'pcfops';
UPDATE 1
uaa=#

i. Confirm that the external_group column now the correct values.

uaa=# select * from external_group_mapping;
               group_id               | external_group |          added          |         origin         | identity_zone_id
--------------------------------------+----------------+-------------------------+------------------------+------------------
 3374447f-d7ec-40ec-878f-87d2bf261ad3 | pcfadmins      | 2018-08-29 02:34:59.361 | external-saml-provider | uaa
 8baf7580-41fa-4e9c-a36e-2b138368388f | pcfadmins      | 2018-08-29 02:34:59.374 | external-saml-provider | uaa
(2 rows)

uaa-#

j. Exit out of the psql client by typing '\q' and pressing Enter.

uaa-# \q
$

k. Restart the tempest-web service to restart the Ops Man authentication system. Then, try to login in Ops Manager UI.

$ sudo service tempest-web restart

 

If the value of the Groups Attribute needs to be corrected in Ops Manager, then follow these steps.

a. Log in to the Ops Manager VM, then log in to the tempest_production database using PSQL client.

$ sudo -u tempest-web psql tempest_production

b. Confirm the current Groups Attribute that is in the 'uaa_configs' table. In this example, the wrong value is 'memberof' and the correct value is 'external_groups'.

tempest_production=# select rbac_saml_groups_attribute from uaa_configs;
 rbac_saml_groups_attribute
----------------------------
 memberof
(1 row)

tempest_production=#

c. Update the table with the correct value.

tempest_production=# UPDATE uaa_configs SET rbac_saml_groups_attribute = 'external_groups' WHERE rbac_saml_groups_attribute = 'memberof';
UPDATE 1
tempest_production=#

d. Confirm that the table now has the correct value.

tempest_production=# select rbac_saml_groups_attribute from uaa_configs;
 rbac_saml_groups_attribute
----------------------------
 external_groups
(1 row)

tempest_production=#

e. Exit out of the psql client by typing '\q' and pressing Enter.

tempest_production-# \q
$

f. Log in to the Ops Manager UAA database using PSQL client.

$ sudo -u tempest-web psql uaa

g. Confirm what is the current SAML Admin Group that is saved in 'external_group_mapping' table.

tempest_production=# select config from identity_provider where type = 'saml';
   config
--------------------------------------------
{"emailDomain":null,"additionalConfiguration":null,"providerDescription":null,"externalGroupsWhitelist":[],"attributeMappings":{"external_groups":"memberof"},"addShadowUserOnLogin":true,"storeCustomAttributes":true,"metaDataLocation":"http://idp.domain/saml2/idp/metadata.php","idpEntityAlias":"external-saml-provider","zoneId":"uaa","nameID":"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress","assertionConsumerIndex":0,"metadataTrustCheck":false,"showSamlLink":true,"linkText":"Log in with organization credentials (SAML)","iconUrl":null,"groupMappingMode":"EXPLICITLY_MAPPED","skipSslValidation":true,"authnContext":null,"socketFactoryClassName":null}
(1 row)

h. Update the table with the correct value.

uaa=# UPDATE identity_provider SET config = '{"emailDomain":null,"additionalConfiguration":null,"providerDescription":null,"externalGroupsWhitelist":[],"attributeMappings":{"external_groups":"external_groups"},"addShadowUserOnLogin":true,"storeCustomAttributes":true,"metaDataLocation":"http://idp.domain/saml2/idp/metadata.php","idpEntityAlias":"external-saml-provider","zoneId":"uaa","nameID":"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress","assertionConsumerIndex":0,"metadataTrustCheck":false,"showSamlLink":true,"linkText":"Log in with organization credentials (SAML)","iconUrl":null,"groupMappingMode":"EXPLICITLY_MAPPED","skipSslValidation":true,"authnContext":null,"socketFactoryClassName":null}' WHERE type = 'saml';
UPDATE 1
uaa=#

i. Confirm that the config column now has the correct value. Specifically check the external_groups value (e.g., {"external_groups":"external_groups"})

uaa=# select config from identity_provider where type = 'saml';
   config
--------------------------------------------
{"emailDomain":null,"additionalConfiguration":null,"providerDescription":null,"externalGroupsWhitelist":[],"attributeMappings":{"external_groups":"external_groups"},"addShadowUserOnLogin":true,"storeCustomAttributes":true,"metaDataLocation":"http://idp.domain/saml2/idp/metadata.php","idpEntityAlias":"external-saml-provider","zoneId":"uaa","nameID":"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress","assertionConsumerIndex":0,"metadataTrustCheck":false,"showSamlLink":true,"linkText":"Log in with organization credentials (SAML)","iconUrl":null,"groupMappingMode":"EXPLICITLY_MAPPED","skipSslValidation":true,"authnContext":null,"socketFactoryClassName":null}
(1 row)

uaa-#

j. Exit out of the psql client by typing '\q' and pressing Enter.

uaa-# \q
$

k. Restart the tempest-web service to restart the Ops Manager authentication system. Then, try to login in Ops Manager UI.

$ sudo service tempest-web restart