Synchronising config-server mirror fails with exception "Username must not be null"
Article ID: 297180
Updated On:
Products
Support Only for Spring
Issue/Introduction
User is unable to synchronize mirror repository of a config-server service instance, and error like "Username must not be null" would be seen in the backend config-server application logs. For example,
2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT ESC[35m[http-nio-8080-exec-2]ESC[0;39m ESC[34mINFO ESC[0;39m i.p.s.c.c.m.RefreshMirrorsEndpoint.refresh - Refreshing git mirrors 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT ESC[35m[http-nio-8080-exec-2]ESC[0;39m ESC[1;31mERRORESC[0;39m o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in c ontext with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Username must not be null] with root cause 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT java.lang.IllegalArgumentException: Username must not be null 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.util.Assert.notNull(Assert.java:201) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.encodeBasicAuth(HttpHeaders.java:1834) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.setBasicAuth(HttpHeaders.java:770) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at org.springframework.http.HttpHeaders.setBasicAuth(HttpHeaders.java:751) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsService.fetchAccessToken(RefreshMirrorsService.java:79) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsService.performRefresh(RefreshMirrorsService.java:47) 2022-11-15T03:54:42.63+0000 [APP/PROC/WEB/0] OUT at io.pivotal.spring.cloud.configserver.management.RefreshMirrorsEndpoint.refresh(RefreshMirrorsEndpoint.java:35)
The credentials used to refresh mirror service is stored in Credhub. And the Credhub path storing the credentials is embedded in backend application environment variables for the config-server service. For example,
$ cf env config-server
Getting env variables for app config-server in org p-spring-cloud-services / space 2c876373-####-####-####-ecfa240644af as admin...
System-Provided:
VCAP_SERVICES: {
"p.mirror-service": [
{
"binding_guid": "2c6ff079-####-####-####-5f2e9780951e",
"binding_name": null,
"credentials": {
"credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials"
},
"instance_guid": "fe359a73-####-####-####-0868bfca5be8",
"instance_name": "mirror-svc",
"label": "p.mirror-service",
"name": "mirror-svc",
"plan": "standard",
"provider": null,
"syslog_drain_url": null,
"tags": [],
"volume_mounts": []
}
]
}
"credhub-ref" field contains the path in Credhub for storing the credentials.
Connect to TAS Credhub by referring to this KB article and retrieve the credential.
$ credhub get -n /c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials -j
{
"id": "608158bc-5b81-44ac-95e5-90e45b9a3308",
"name": "/c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials",
"type": "json",
"value": {
"git": {
"basedir": "/home/vcap/app/config-repo/default",
"ignoreLocalSshSettings": true,
......
"refresh_mirrors": {
"enabled": true
}
},
"metadata": null,
"version_created_at": "2022-11-10T07:24:39Z"
}
As you can see that the "refresh_mirrors" section only contains "enabled" field. But a complete record should also include client credentials for getting token from UAA as shown in example below.
"refresh_mirrors": {
"access_token_uri": "https://uaa.example.com:443/oauth/token",
"client_id": "mirror_client_8f303ab3-####-####-####-98fd-73bc2da0e459",
"client_secret": "a3c4ighkp0bf7",
"enabled": true,
"url": "https://scs-mirror-service.run.example.com/actuator/refreshmirrors/8f303ab3-####-####-####-73bc2da0e459"
}
The client credentials should be injected to Credhub at the time of creating the config-server service instance. However it's unknown why it's missing in this case.
Environment
Product Version: 3.1
Resolution
The issue could be resolved by resetting client credentials in Credhub.
1) Log into TAS UAA endpoint using UAA CLI (UAAC)
2) Check if the mirror client still exists in UAA or not. For example,
3) If the mirror client is still existing in UAA, then reset the client secret with command uaac secret set <mirror client_id> -s <new secret>.
If the mirror client doesn't exist in UAA any more, run uaac client add command to add a new client with desired ID. For example,
4) Log into TAS Credhub by referring to this KB article and export all credentials from Credhub as a backup
6) Update the copied file from step 5 and add the missing properties (access_token_uri, client_id, client_secret, url) to "refresh_mirrors" section. For example,
7) Delete the credential from Credhub
1) Log into TAS UAA endpoint using UAA CLI (UAAC)
2) Check if the mirror client still exists in UAA or not. For example,
$ uaac client get mirror_client_fe359a73-####-####-####-0868bfca5be8 scope: uaa.none client_id: mirror_client_fe359a73-####-####-####-0868bfca5be8 resource_ids: none authorized_grant_types: client_credentials autoapprove: access_token_validity: 360 authorities: mirror_service_fe359a73-####-####-####-0868bfca5be8.refresh lastmodified: 1603968497000The client ID is something like mirror_client_<config-server service instance_guid>.
3) If the mirror client is still existing in UAA, then reset the client secret with command uaac secret set <mirror client_id> -s <new secret>.
If the mirror client doesn't exist in UAA any more, run uaac client add command to add a new client with desired ID. For example,
$ uaac client add --name mirror_client_fe359a73-####-####-####-0868bfca5be8 --authorized_grant_types client_credentials --authorities mirror_service_fe359a73-####-####-####-0868bfca5be8.refresh --access_token_validity 360 --secret abcd1234 mirror_client_fe359a73-####-####-####-0868bfca5be8 scope: uaa.none client_id: mirror_client_fe359a73-####-####-####-0868bfca5be8 resource_ids: none authorized_grant_types: client_credentials autoapprove: access_token_validity: 360 authorities: mirror_service_fe359a73-####-####-####-0868bfca5be8.refresh name: mirror_client_fe359a73-####-####-####-0868bfca5be8 required_user_groups: lastmodified: 1674449478000 id: mirror_client_fe359a73-####-####-####-0868bfca5be8
4) Log into TAS Credhub by referring to this KB article and export all credentials from Credhub as a backup
$ credhub export -f export-credhub.out5) Make a copy of the exported file from above step and remove all other credentials except the one we want to update the missing fields like client_id and client_secret. For example, only keep section /c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials.
6) Update the copied file from step 5 and add the missing properties (access_token_uri, client_id, client_secret, url) to "refresh_mirrors" section. For example,
credentials:
- name: /c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials
type: json
value:
git:
......
refresh_mirrors:
access_token_uri: https://uaa.<system domain>:443/oauth/token
client_id: mirror_client_fe359a73-####-####-####-0868bfca5be8
client_secret: abcd1234
enabled: true
url: https://scs-mirror-service.<system domain>/actuator/refreshmirrors/fe359a73-####-####-####-0868bfca5be8
7) Delete the credential from Credhub
$ credhub delete -n /c/p.spring-cloud-services-scs-mirror-service/fe359a73-####-####-####-0868bfca5be8/credentials8) Import the credential from file prepared in step 6
$ credhub import -f <file from step 6>9) Restart backend app config-server for the problematic config-server service instance
$ cf t -o p-spring-cloud-services -s fe359a73-####-####-####-0868bfca5be8 $ cf restart config-server
Feedback
Yes
No
Powered by
