Password rotation for [email protected] causes issues when multiple VMware Cloud Foundation instances share a single SSO domain

Products

• VMware Cloud Foundation

Issue/Introduction

• This article provides information about the impact of changing the SSO administrator password  "[email protected]" .
• The article also provides steps to remediate the issue through steps to update the  "[email protected]" on the second instance.

Symptoms:

• Multiple VMware Cloud Foundation instances in the same SSO Domain.
• SSO Administrator password "[email protected]" rotation (manually or scheduled) on one of the VMware Cloud Foundation instances.
• SDDC manager in second instance becomes inaccessible with  "[email protected]" or any other privileged user.
• SDDC manager database in the second VMware Cloud Foundation Instance has incorrect password for "[email protected]" .

Cause

Multiple VMware Cloud Foundation instances can be joined together in a single SSO domain and each VMware Cloud Foundation instance has an entry for "[email protected]" in the associated SDDC Manager database.
Since all VCF instances are joined together in same PSC SSO, the credentials for SSO account  "[email protected]" needs to be manually updated in each additional VMware Cloud Foundation instance after a password rotation in the primary instance.

The credential for  "[email protected]"  can be rotated and/or updated from the primary VMware Cloud Foundation instance, which results in additional instances having a stale password which renders the SDDC Manager UI inaccessible.
This requires a Remediate operation in the additional VMware Cloud Foundation instances immediately so that password stored in secondary instances is updated to reflect the change.

Resolution

To update the SSO Administrator Password on an additional instance, follow the below steps:

1. On the primary VMware Cloud Foundation instance, connect to SDDC Manager via ssh.
2. su to root
3. Run the lookup_passwords command using admin@local account or any other SSO account which has the ADMIN role in SDDC Manager, and note down the password for "[email protected]"

vcf@sddc-manager [ ~ ]$ lookup_passwords -e PSC

Password lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Operations and Administration Guide for setting up ADMIN user.

Enter page number (optional):
Enter page size (optional, default=50):
Enter Username: [email protected]
Enter Password:
        PSC
        identifiers: x.x.x.x,vcenter-1.vrack.vsphere.local
        workload: sddcId-1001
                username: [email protected]
                password: *********     --->New Password 
                type: SSO
                account type: SYSTEM

Note: X.X.X.X is the vCenter Server IP address
 

2. On the additional VMware Cloud Foundation instance, connect to SDDC Manager via ssh.
3. su to root
4. Using admin@local account or any other SSO account which has the ADMIN role in SDDC Manager, create an API access_token

vcf@sddc-manager [ ~ ]$ curl -d '{"username" : "[email protected]", "password" : "********"}' -H "Content-Type: application/json" -X POST localhost/v1/tokens -k | jq
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 1798 0 1727 100 71 5588 229 --:--:-- --:--:-- --:--:-- 5818
{
  "accessToken": "eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI5ZGYxNzU0Yi04ZDc3LTQzODMtYTEyZC0wOTdkMzJiZTEzMGQiLCJpYXQiOjE2MjczODIyODksInN1YiI6InZjZi1zZWN1cmUtdXNlckB2c3BoZXJlLmxvY2FsIiwiaXNzIjoidmNmLWF1dGgiLCJhdWQiOiJzZGRjLXNlcnZpY2VzIiwibmJmIjoxNjI3MzgyMjg5LCJleHAiOjE2MjczODU4ODksInVzZXIiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsIm5hbWUiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsInNjb3BlIjpbIlNERENfRkVERVJBVElPTl9XUklURSIsIkFWTl9XUklURSIsIlNERENfTUFOQUdFUl9SRUFEIiwiQ0VSVF9XUklURSIsIkNPTVBPU0FCSUxJVFlfV1JJVEUiLCJMSUNFTlNFX0tFWV9SRUFEIiwiQ09NUE9TQUJJTElUWV9SRUFEIiwiRURHRV9DTFVTVEVSX1dSSVRFIiwiVVNFUl9SRUFEIiwiQ1JFREVOVElBTF9XUklURSIsIkJBQ0tVUF9DT05GSUdfUkVBRCIsIkNMVVNURVJfV1JJVEUiLCJBVk5fUkVBRCIsIlZBU0FfUFJPVklERVJfUkVBRCIsIkRPTUFJTl9XUklURSIsIkNFSVBfUkVBRCIsIlNPU19XUklURSIsIlNERENfTUFOQUdFUl9XUklURSIsIk5UUF9XUklURSIsIkRFUE9UX0NPTkZJR19XUklURSIsIkRFUE9UX0NPTkZJR19SRUFEIiwiSE9TVF9XUklURSIsIkJBQ0tVUF9SRVNUT1JFX1JFQUQiLCJDRVJUX1JFQUQiLCJVU0VSX1dSSVRFIiwiVVBHUkFERV9SRUFEIiwiT1RIRVJfUkVBRCIsIlNPU19SRUFEIiwiU0VDVVJJVFlfQ09ORklHX1JFQUQiLCJDUkVERU5USUFMX1JFQUQiLCJIT1NUX1JFQUQiLCJDRUlQX1dSSVRFIiwiT1RIRVJfV1JJVEUiLCJMSUNFTlNFX0tFWV9XUklURSIsIkNBX1JFQUQiLCJORVRXT1JLX1BPT0xfV1JJVEUiLCJXQ1BfUkVBRCIsIkJBQ0tVUF9SRVNUT1JFX1dSSVRFIiwiTlRQX1JFQUQiLCJFREdFX0NMVVNURVJfUkVBRCIsIkJBQ0tVUF9DT05GSUdfV1JJVEUiLCJXQ1BfV1JJVEUiLCJTRVJWSUNFX0FDQ09VTlRfV1JJVEUiLCJORVRXT1JLX1BPT0xfUkVBRCIsIkNBX1dSSVRFIiwiQ0xVU1RFUl9SRUFEIiwiVkFTQV9QUk9WSURFUl9XUklURSIsIkROU19XUklURSIsIlZSU0xDTV9XUklURSIsIkROU19SRUFEIiwiU0VSVklDRV9BQ0NPVU5UX1JFQUQiLCJTRERDX0ZFREVSQVRJT05fUkVBRCIsIkRPTUFJTl9SRUFEIiwiVlJTTENNX1JFQUQiLCJVUEdSQURFX1dSSVRFIl0sInJvbGUiOlsiQURNSU4iXX0.w5H0W8NhbTwXafrsWJ0dNBg4WphsENoqO5ERLpGXZVQ",
  "refreshToken": {
    "id": "37f118d1-1264-4e65-b52a-9b763f2f582a"
  }
}
vcf@sddc-manager [ ~ ]$
vcf@sddc-manager [ ~ ]$

Set access token from above response to a variable access_token:

vcf@sddc-manager [ ~ ]$ access_token=eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI5ZGYxNzU0Yi04ZDc3LTQzODMtYTEyZC0wOTdkMzJiZTEzMGQiLCJpYXQiOjE2MjczODIyODksInN1YiI6InZjZi1zZWN1cmUtdXNlckB2c3BoZXJlLmxvY2FsIiwiaXNzIjoidmNmLWF1dGgiLCJhdWQiOiJzZGRjLXNlcnZpY2VzIiwibmJmIjoxNjI3MzgyMjg5LCJleHAiOjE2MjczODU4ODksInVzZXIiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsIm5hbWUiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsInNjb3BlIjpbIlNERENfRkVERVJBVElPTl9XUklURSIsIkFWTl9XUklURSIsIlNERENfTUFOQUdFUl9SRUFEIiwiQ0VSVF9XUklURSIsIkNPTVBPU0FCSUxJVFlfV1JJVEUiLCJMSUNFTlNFX0tFWV9SRUFEIiwiQ09NUE9TQUJJTElUWV9SRUFEIiwiRURHRV9DTFVTVEVSX1dSSVRFIiwiVVNFUl9SRUFEIiwiQ1JFREVOVElBTF9XUklURSIsIkJBQ0tVUF9DT05GSUdfUkVBRCIsIkNMVVNURVJfV1JJVEUiLCJBVk5fUkVBRCIsIlZBU0FfUFJPVklERVJfUkVBRCIsIkRPTUFJTl9XUklURSIsIkNFSVBfUkVBRCIsIlNPU19XUklURSIsIlNERENfTUFOQUdFUl9XUklURSIsIk5UUF9XUklURSIsIkRFUE9UX0NPTkZJR19XUklURSIsIkRFUE9UX0NPTkZJR19SRUFEIiwiSE9TVF9XUklURSIsIkJBQ0tVUF9SRVNUT1JFX1JFQUQiLCJDRVJUX1JFQUQiLCJVU0VSX1dSSVRFIiwiVVBHUkFERV9SRUFEIiwiT1RIRVJfUkVBRCIsIlNPU19SRUFEIiwiU0VDVVJJVFlfQ09ORklHX1JFQUQiLCJDUkVERU5USUFMX1JFQUQiLCJIT1NUX1JFQUQiLCJDRUlQX1dSSVRFIiwiT1RIRVJfV1JJVEUiLCJMSUNFTlNFX0tFWV9XUklURSIsIkNBX1JFQUQiLCJORVRXT1JLX1BPT0xfV1JJVEUiLCJXQ1BfUkVBRCIsIkJBQ0tVUF9SRVNUT1JFX1dSSVRFIiwiTlRQX1JFQUQiLCJFREdFX0NMVVNURVJfUkVBRCIsIkJBQ0tVUF9DT05GSUdfV1JJVEUiLCJXQ1BfV1JJVEUiLCJTRVJWSUNFX0FDQ09VTlRfV1JJVEUiLCJORVRXT1JLX1BPT0xfUkVBRCIsIkNBX1dSSVRFIiwiQ0xVU1RFUl9SRUFEIiwiVkFTQV9QUk9WSURFUl9XUklURSIsIkROU19XUklURSIsIlZSU0xDTV9XUklURSIsIkROU19SRUFEIiwiU0VSVklDRV9BQ0NPVU5UX1JFQUQiLCJTRERDX0ZFREVSQVRJT05fUkVBRCIsIkRPTUFJTl9SRUFEIiwiVlJTTENNX1JFQUQiLCJVUEdSQURFX1dSSVRFIl0sInJvbGUiOlsiQURNSU4iXX0.w5H0W8NhbTwXafrsWJ0dNBg4WphsENoqO5ERLpGXZVQ
 

3. On the additional VMware Cloud Foundation instance, perform a credentials lookup using REST API

vcf@sddc-manager [ ~ ]$ curl localhost/v1/credentials?resourceType=PSC -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 531 0 531 0 0 2794 0 --:--:-- --:--:-- --:--:-- 2794
{
  "elements": [
    {
      "id": "8491d072-c0eb-4fea-a594-7f4b9253dc97",
      "credentialType": "SSO",
      "accountType": "SYSTEM",
      "username": "[email protected]",
      "password": "**********",   -->old SSO Admin Password
      "creationTimestamp": "2021-07-21T22:30:59.760Z",
      "modificationTimestamp": "2021-07-21T22:30:59.760Z",
      "resource": {
        "resourceId": "e48b6ad3-4256-4156-a795-99b957c943b0",
        "resourceName": "vcenter-1.vrack.vsphere.local",
        "resourceIp": "x.x.x.x",  --->vCenter IP
        "resourceType": "PSC",
        "domainName": "sddcId-1001"
      }
    }
  ],
  "pageMetadata": {
    "pageNumber": 0,
    "pageSize": 1,
    "totalElements": 1,
    "totalPages": 1
  }
}
vcf@sddc-manager [ ~ ]$

4. On the additional VMware Cloud Foundation instance, perform a REMEDIATE operation, providing the json data from the result of step 3, but the new password from primary instance. The response will have task Id.

vcf@sddc-manager [ ~ ]$ cat remediate.json
{
  "operationType" : "REMEDIATE",
  "elements" : [ {
    "resourceName" : "vcenter-1.vrack.vsphere.local",
    "resourceType" : "PSC",
    "credentials" : [ {
      "username" : "[email protected]",
      "password" : "***********"  --> New Password
    } ]
  } ]
}

vcf@sddc-manager [ ~ ]$ curl localhost/v1/credentials -X PATCH -[email protected] -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq
% Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 315 0 68 100 247 84 305 --:--:-- --:--:-- --:--:-- 389
{
  "id": "bbd90419-99cc-4677-a5dd-927af03d3bab",
  "status": "IN_PROGRESS"
}

5.  On the additional VMware Cloud Foundation instance, monitor task status by using the task id. Please refer to the VMware Cloud Foundation Rest API documentation.

*Once the status is successful, VCF secondary instance is now synced and SDDC Manager UI should start working.

 

vcf@sddc-manager [ ~ ]$ curl localhost/v1/tasks/bbd90419-99cc-4677-a5dd-927af03d3bab -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq
% Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 841 0 841 0 0 17893 0 --:--:-- --:--:-- --:--:-- 17893
{
  "id": "bbd90419-99cc-4677-a5dd-927af03d3bab",
  "name": "Credentials remediate operation",
  "status": "SUCCESSFUL",
  "creationTimestamp": "2021-07-27T10:38:39.007Z",
  "subTasks": [
    {
      "name": "Password remediate prevalidation",
      "description": "Prevalidation of password remediate request",
      "status": "SUCCESSFUL",
      "creationTimestamp": "2021-07-27T10:38:39.007Z",
      "completionTimestamp": "2021-07-27T10:38:39.007Z"
    },
    {
      "name": "Password remediate for resource : vcenter-1.vrack.vsphere.local, user : [email protected] and credential type : SSO",
      "description": "Password remediate for resource : vcenter-1.vrack.vsphere.local, user : [email protected] and credential type : SSO",
      "status": "SUCCESSFUL",
      "creationTimestamp": "2021-07-27T10:38:40.090Z",
      "completionTimestamp": "2021-07-27T10:38:40.090Z"
    }
  ],
  "resolutionStatus": "UNRESOLVED",
  "isCancellable": false
}