Manually synchronized mirror of config-server instance got "cannot deserialize instance of `java.lang.String` out of START_OBJECT token"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When trying to refresh a config-server service instance, Git mirrors manually uses the cf CLI or "Synchronize Mirrors" button on the service instance dashboard and it fails with an error similar to the one below:

FAILED Authenticated post to 'https://config-server-97d1bf3c-d06d-4f26-a466-c4c2d1659862.app.example.com/actuator/refreshmirrors' failed: 500 Internal Server Error

And more errors could be seen in logs of apps for the config-server service instance. For example:

2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor.beforeInvocation - Secure object: FilterInvocation: URL: /actuator/refreshmirrors; Attributes: [hasAuthority('SCOPE_cloud_controller.admin') || hasRole('cloud_controller.admin') || @spaceDeveloperChecker.check(authentication)]
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor.authenticateIfRequired - Previously Authenticated: org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken@966a9041: Principal: org.springframework.security.oauth2.jwt.Jwt@8fcbf6bc; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@3bcc: RemoteIpAddress: 10.0.0.0; SessionId: null; Granted Authorities: SCOPE_cloud_controller.read, SCOPE_actuator.read, SCOPE_cloud_controller.admin_read_only, SCOPE_cloud_controller.user, SCOPE_cloud_controller.write, SCOPE_openid, SCOPE_doppler.firehose, SCOPE_cloud_controller.admin
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor.beforeInvocation - Authorization successful
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor.beforeInvocation - RunAsManager did not change Authentication object
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.servlet.DispatcherServlet.traceDebug - POST "/actuator/refreshmirrors", parameters={}
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] INFO  i.p.s.c.c.m.RefreshMirrorsEndpoint.refresh - Refreshing git mirrors
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - HTTP POST https://uaa.sys.example.com:443/oauth/token
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Accept=[application/json, application/*+json]
   2020-09-11T09:09:14.99-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Writing [{grant_type=[client_credentials], client_id=[mirror_client_xxxx], client_secret=xxxx}] with org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter
   2020-09-11T09:09:15.05-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Response 200 OK
   2020-09-11T09:09:15.05-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Reading to [io.pivotal.spring.cloud.configserver.management.RefreshMirrorsService$TokenResponse]
   2020-09-11T09:09:15.05-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - HTTP POST https://scs-mirror-service.sys.example.com/actuator/refreshmirrors/a3c09520-d063-44ca-bb66-24b4a57f5be0
   2020-09-11T09:09:15.05-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Accept=[application/json, application/*+json]
   2020-09-11T09:09:15.05-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Writing [{}] as "application/json"
   2020-09-11T09:09:15.34-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Response 200 OK
   2020-09-11T09:09:15.34-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.client.RestTemplate.debug - Reading to [java.util.Map<java.lang.String, java.lang.String>]
   2020-09-11T09:09:15.34-0400 [APP/PROC/WEB/1] OUT [http-nio-8080-exec-10] DEBUG o.s.web.servlet.DispatcherServlet.logResult - Failed to complete request: org.springframework.web.client.RestClientException: Error while extracting response for type [java.util.Map<java.lang.String, java.lang.String>] and content type [application/json]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token

Resolution

Starting from SCS 3.1.15, the response body format of the /actuator/refreshmirrors request was changed. However if some SCS service instances were created on SCS 3.x (<3.1.15), they would be unable to understand the new format of the response body and then trigger the above errors. 

Actually refreshing the mirror was successful as the status code for POST request is 200. However config-server instance with an old SCS 3.x version just could not decode the response data correctly.

Run the following command to confirm version info of the config-server service instance in question. For example:

$ curl https://config-server-97d1bf3c-d06d-4f26-a466-c4c2d1659862.app.example.com/actuator/info -k
{"git":{"commit":{"time":"2020-06-10T08:20:59Z","id":"bcef0e2"},"branch":"HEAD"},"build":{"version":"3.1.12","artifact":"config-server","name":"config-server","group":"io.pivotal.spring.cloud","time":"2020-06-10T15:43:30.411Z"}}

If the SCS tile version is 3.1.15 but the config-server instance version is older, then upgrade the config-server instance to latest version with following command:

$ cf update-service <config-sever instance name> -c '{"upgrade": true}'