Mirror service fails to refresh with "Invalid username or password" error message.

Products

• Support Only for Spring

Issue/Introduction

This document focuses on the scenario in which there are multiple service instances (SI) sharing the same GIT repository and its credentials have recently been changed, leading the mirror logs to display the "Invalid username or password" error when it is performing the periodic refresh to the repositories.


The mirror-service.stdout.log file (located at /var/vcap/sys/log/mirror-service) logs show the following:

mirror-service.stdout.log,
 

2023-04-25 19:35:22.715  INFO 6 --- [   scheduling-1] i.p.s.c.m.refresh.ScheduledRefresh       : Refreshing all mirrors (periodic)
2023-04-25 19:35:22.980  INFO 6 --- [   scheduling-1] i.p.s.c.m.refresh.RefreshExecutor        : Updating mirror repo with source uri https://rbcgithub.fg.rbc.com/UBC0/Config-Repo-Microservice
2023-04-25 19:35:22.981  INFO 6 --- [   scheduling-1] i.p.s.c.mirrorservice.mirrors.ShellUtil  : Running shell command: git remote set-url origin https://[REDACTED:REDACTED]@company.com/UBC0/Config-Repo-Microservice
2023-04-25 19:35:22.983  INFO 6 --- [   scheduling-1] i.p.s.c.mirrorservice.mirrors.ShellUtil  : Running shell command: git -c http.sslVerify=false fetch origin
2023-04-25 19:35:25.930  INFO 6 --- [   scheduling-1] i.p.s.c.mirrorservice.mirrors.ShellUtil  : remote: Invalid username or password.
2023-04-25 19:35:25.930  INFO 6 --- [   scheduling-1] i.p.s.c.mirrorservice.mirrors.ShellUtil  : fatal: Authentication failed for 'https://company.com/UBC0/Config-Repo-Microservice/'
2023-04-25 19:35:25.930 ERROR 6 --- [   scheduling-1] i.p.s.c.m.refresh.ScheduledRefresh       : Failed to perform periodic refresh on mirror: /var/vcap/store/mirror/b39a96d8c1262e330a7b71d8b069989d/Config-Repo-Microservice

config-server backend app logs will trigger 401 errors for /actuator/refreshmirrors actuator endpoint:
 

959ce2e8-b0ca-44bf-831b-490dc662e916 RTR/12 2023-04-11T21:09:16.350976566Z OUT config-server-dd16de4d-3736-4e73-96f9-cfb92a190852.apps.cf2.devfg.company.com - [2023-04-11T21:09:16.343542580Z] "POST /actuator/refreshmirrors HTTP/1.1" 401 0 0 "https://config-server-dd16de4d-3736-4e73-96f9-cfb92a190852.apps.cf2.devfg.company.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" "123.123.123.39:10755" "10.14.252.254:61001" x_forwarded_for:"123.123.123.39" x_forwarded_proto:"https" vcap_request_id:"051f2871-4020-431b-7aeb-f908bc5f4276" response_time:0.007146 gorouter_time:0.000345 app_id:"959ce2e8-b0ca-44bf-831b-490dc662e916" app_index:"2" instance_id:"af42f310-2337-4712-7a59-7957" x_cf_routererror:"-" x_b3_traceid:"7121a7337ce4ef761787a9fc7f7e1ccd" x_b3_spanid:"1787a9fc7f7e1ccd" x_b3_parentspanid:"-" b3:"7121a7337ce4ef761787a9fc7f7e1ccd-1787a9fc7f7e1ccd"

Environment

Product Version: Other

Resolution

This is a feature and limitation that comes with mirror-service .

- It creates a single mirror for each git repository, regardless of the number of SI using that repository. Assuming you have N service instances using the same git repository, they will share a single mirror (clone) of that repository but they can/might have different credentials (provided during instance creating or update).

- When a user performs the manual sync/refresh for a particular service instance, the credentials configured for that particular SI will be used to fetch the remote repository. If the credentials are correct the fetch will be successful, otherwise it fails.

- During periodic refresh by mirror service, that mirror will be refreshed only once, using the credentials of one of the service instances. If the chosen credentials happen to be the wrong one the refresh will fail, even if all other (N - 1) credentials are correct. 

- The cf update command cannot be used to change Git credentials. In order to fix the issue, if you need to update or change GIT credentials delete the config-server instance and recreated it with the new credentials, if using multiple SI sharing the same repo, they will all have to be deleted and recreated with the new credentials.

- Based on the above explanation, if only a few SI are recreated the issue might persist or become intermittent until all the SI are properly recreated.