Synchronising config-server mirror fails with exception "Username must not be null"

Products

• Support Only for Spring

Issue/Introduction

User is unable to synchronize mirror repository of a config-server service instance, and error like "Username must not be null" would be seen in the backend config-server application logs. For example,

2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT ESC[35m[http-nio-8080-exec-2]ESC[0;39m ESC[34mINFO ESC[0;39m i.p.s.c.c.m.RefreshMirrorsEndpoint.refresh - Refreshing git mirrors
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT ESC[35m[http-nio-8080-exec-2]ESC[0;39m ESC[1;31mERRORESC[0;39m o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in c
ontext with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Username must not be null] with root cause
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT java.lang.IllegalArgumentException: Username must not be null
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.util.Assert.notNull(Assert.java:201)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.encodeBasicAuth(HttpHeaders.java:1834)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.setBasicAuth(HttpHeaders.java:770)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.setBasicAuth(HttpHeaders.java:751)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsService.fetchAccessToken(RefreshMirrorsService.java:79)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsService.performRefresh(RefreshMirrorsService.java:47)
   2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsEndpoint.refresh(RefreshMirrorsEndpoint.java:35)

The credentials used to refresh mirror service is stored in Credhub. And the Credhub path storing the credentials is embedded in backend application environment variables for the config-server service. For example,

$ cf env config-server
Getting env variables for app config-server in org p-spring-cloud-services / space 2c876373-cfcd-4b60-b4b3-ecfa240644af as admin...
System-Provided:
VCAP_SERVICES: {
  "p.mirror-service": [
    {
      "binding_guid": "2c6ff079-cfb2-42c6-a467-5f2e9780951e",
      "binding_name": null,
      "credentials": {
        "credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/fe359a73-4c7a-4df2-a065-0868bfca5be8/credentials"
      },
      "instance_guid": "fe359a73-4c7a-4df2-a065-0868bfca5be8",
      "instance_name": "mirror-svc",
      "label": "p.mirror-service",
      "name": "mirror-svc",
      "plan": "standard",
      "provider": null,
      "syslog_drain_url": null,
      "tags": [],
      "volume_mounts": []
    }
  ]
}

"credhub-ref" field contains the path in Credhub for storing the credentials.

Connect to TAS Credhub by referring to this KB article and retrieve the credential.

$ credhub get -n /c/p.spring-cloud-services-scs-mirror-service/2c876373-cfcd-4b60-b4b3-ecfa240644af/credentials -j
{
        "id": "608158bc-5b81-44ac-95e5-90e45b9a3308",
        "name": "/c/p.spring-cloud-services-scs-mirror-service/fe359a73-4c7a-4df2-a065-0868bfca5be8/credentials",
        "type": "json",
        "value": {
                "git": {
                        "basedir": "/home/vcap/app/config-repo/default",
                        "ignoreLocalSshSettings": true,
                ......
                "refresh_mirrors": {
                        "enabled": true
                }
        },
        "metadata": null,
        "version_created_at": "2022-11-10T07:24:39Z"
}

As you can see that the "refresh_mirrors" section only contains "enabled" field. But a complete record should also include client credentials for getting token from UAA as shown in example below.

 "refresh_mirrors": {
      "access_token_uri": "https://uaa.example.com:443/oauth/token",
      "client_id": "mirror_client_8f303ab3-917a-4152-98fd-73bc2da0e459",
      "client_secret": "a3c4ighkp0bf7",
      "enabled": true,
      "url": "https://scs-mirror-service.run.example.com/actuator/refreshmirrors/8f303ab3-917a-4152-98fd-73bc2da0e459"
    }

The client credentials should be injected to Credhub at the time of creating the config-server service instance. However it's unknown why it's missing in this case.

Environment

Product Version: 3.1

Resolution

The issue could be resolved by resetting client credentials in Credhub. 

1) Log into TAS UAA endpoint using UAA CLI (UAAC)

2) Check if the mirror client still exists in UAA or not. For example,

$ uaac client get mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
  scope: uaa.none
  client_id: mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
  resource_ids: none
  authorized_grant_types: client_credentials
  autoapprove:
  access_token_validity: 360
  authorities: mirror_service_fe359a73-4c7a-4df2-a065-0868bfca5be8.refresh
  lastmodified: 1603968497000

The client ID is something like mirror_client_<config-server service instance_guid>. 

3) If the mirror client is still existing in UAA, then reset the client secret with command uaac secret set <mirror client_id> -s <new secret>. 
If the mirror client doesn't exist in UAA any more, run uaac client add command to add a new client with desired ID.  For example,

$ uaac client add --name mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8 --authorized_grant_types client_credentials  --authorities mirror_service_fe359a73-4c7a-4df2-a065-0868bfca5be8.refresh --access_token_validity 360 --secret abcd1234  mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
  scope: uaa.none
  client_id: mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
  resource_ids: none
  authorized_grant_types: client_credentials
  autoapprove:
  access_token_validity: 360
  authorities: mirror_service_fe359a73-4c7a-4df2-a065-0868bfca5be8.refresh
  name: mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
  required_user_groups:
  lastmodified: 1674449478000
  id: mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8

4) Log into TAS Credhub by referring to this KB article and export all credentials from Credhub as a backup

$ credhub export -f export-credhub.out 

5) Make a copy of the exported file from above step and remove all other credentials except the one we want to update the missing fields like client_id and client_secret. For example, only keep section /c/p.spring-cloud-services-scs-mirror-service/fe359a73-4c7a-4df2-a065-0868bfca5be8/credentials. 

6) Update the copied file from step 5 and add the missing properties (access_token_uri, client_id, client_secret, url) to "refresh_mirrors" section. For example,

credentials:
- name: /c/p.spring-cloud-services-scs-mirror-service/fe359a73-4c7a-4df2-a065-0868bfca5be8/credentials
  type: json
  value:
    git:
......
      refresh_mirrors:
        access_token_uri: https://uaa.<system domain>:443/oauth/token
        client_id: mirror_client_fe359a73-4c7a-4df2-a065-0868bfca5be8
        client_secret: abcd1234
        enabled: true
        url: https://scs-mirror-service.<system domain>/actuator/refreshmirrors/fe359a73-4c7a-4df2-a065-0868bfca5be8

7)  Delete the credential from Credhub

$ credhub delete -n /c/p.spring-cloud-services-scs-mirror-service/fe359a73-4c7a-4df2-a065-0868bfca5be8/credentials

8) Import the credential from file prepared in step 6

$ credhub import -f <file from step 6>

9) Restart backend app config-server for the problematic config-server service instance

$ cf t -o p-spring-cloud-services -s fe359a73-4c7a-4df2-a065-0868bfca5be8
$ cf restart config-server