SCS Config Server Label Not Found

Products

Support Only for Spring

Issue/Introduction

This Knowledge Base (KB) article details a reason for why a Spring Cloud Services (SCS) config-server Service Instance (SI) may not be able to find its configured git label.

A config-server is a central component that manages configuration settings for applications, promoting consistency and easy updates across the system. More details about SCS config-servers can be found in the documentation.

When setting up a config-server with Git, you specify various configuration details. This includes an external git repository URI, which allows the SCS mirror service to clone the target repo to the local mirror. You also define a Git label, which represents the target branch from the external repo to configure the config-server with. If a config-server is created with a label not found in the local mirror, an error is thrown (e.g., using the label "develop").

 2023-11-20T11:54:33.74-0500 [APP/PROC/WEB/0] OUT [http-nio-8080-exec-8] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.cloud.config.server.environment.NoSuchLabelException: No such label: develop] with root cause
 2023-11-20T11:54:33.74-0500 [APP/PROC/WEB/0] OUT org.eclipse.jgit.api.errors.RefNotFoundException: Ref develop cannot be resolved

If the label configured for the config-server exists in the upstream git repo, but the mirror service does not have it in the local mirror, then the config-server may just need its mirrors refreshed to fetch the new labels. This is an expected behavior due to the internal mechanics that the mirror service uses to manage its local mirrors.

Environment

Product Version: 3.1

Resolution

In order to understand why this would happen, a high level understanding of mirror service is required. The mirror service is hosted on the SCS VM and local mirrors are saved to the persistent disk at the following location:

/var/vcap/store/mirror/<mirror-ID>/<repository>/

When a config-server configured with git is created, mirror service considers if it already has a local mirror for the provided external git repository URI.

If mirror service discovers it does not have a local mirror for the provided repo, then it will create a new one as indicated from its logs:

2023-11-20T16:38:23.439Z INFO 5 --- [ctor-http-nio-4] s.c.m.b.DelegatingServiceInstanceService : Creating service instance with id 2938e7d1-0257-4b11-82d0-5fe265bf3517
2023-11-20T16:38:23.442Z INFO 5 --- [ctor-http-nio-4] i.p.s.c.m.broker.MirrorManager : Creating new mirror for [email protected]:pivotal-Josh-Gainey/my-scs-test-repo
2023-11-20T16:38:23.442Z INFO 5 --- [ctor-http-nio-4] i.p.s.c.m.mirrors.ShellSshMirrorServer : Creating mirror 55ff81b3084dc342f02cd7a74a4c0155/my-scs-test-repo
2023-11-20T16:38:23.443Z INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil : Running shell command: git -c "core.sshCommand=ssh -i /var/vcap/store/mirror/55ff81b3084dc342f02cd7a74a4c0155/private-key -o StrictHostKeyChecking=no" clone --mirror [email protected]:pivotal-Josh-Gainey/my-scs-test-repo /var/vcap/store/mirror/55ff81b3084dc342f02cd7a74a4c0155/my-scs-test-repo
2023-11-20T16:38:24.980Z INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil : Cloning into bare repository '/var/vcap/store/mirror/55ff81b3084dc342f02cd7a74a4c0155/my-scs-test-repo'...

If mirror service discovers it does have a local mirror for the provided repo, then it will reuse the existing mirror as indicated from its logs:

2023-11-20T16:42:05.462Z  INFO 5 --- [ctor-http-nio-2] s.c.m.b.DelegatingServiceInstanceService : Creating service instance with id aa901d63-75b6-4f8b-a4b3-e9543ccfbbe6
2023-11-20T16:42:05.467Z  INFO 5 --- [ctor-http-nio-2] i.p.s.c.m.broker.MirrorManager           : Mirror for [email protected]:pivotal-Josh-Gainey/my-scs-test-repo already exists

Based on the above information, the following sequence of events can result in the scenario where a label configured for a config-server exists in the upstream git repo, but the mirror service does not have it in the local mirror:

1 - Create config-server-A pointing to repo-A branch branch-A and no periodic refresh.
2 - Create branch-B in repo-A in the external git repo.
3 - Create config-server-B pointing to repo-A branch branch-B and no periodic refresh.

When config-server-B is created, the mirror service will realize it already has a mirror for repo-A. If the local mirror on the SCS VM for this repo has not been refreshed since config-server-A's creation, then the mirror will not have branch-B and config-server-B will face errors until the mirror is refreshed. Refreshing the mirror will result in a "git fetch" command being ran, which will then pull in the new branch.

After refreshing the config-server mirror we can the new branch is now discovered:

2023-11-20T20:44:54.545Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.m.refresh.RefreshEndpoint        : Refreshing Git mirrors for service instance with id aa901d63-75b6-4f8b-a4b3-e9543ccfbbe6
2023-11-20T20:44:54.794Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.m.refresh.RefreshExecutor        : Updating mirror repo with source uri [email protected]:pivotal-Josh-Gainey/my-scs-test-repo
2023-11-20T20:44:54.797Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil  : Running shell command: git -c "core.sshCommand=ssh -i /var/vcap/store/mirror/55ff81b3084dc342f02cd7a74a4c0155/private-key -o StrictHostKeyChecking=no" fetch origin
2023-11-20T20:44:56.340Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil  : From github.com:pivotal-Josh-Gainey/my-scs-test-repo
2023-11-20T20:44:56.340Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil  :  * [new branch]      develop    -> develop
2023-11-20T20:44:56.340Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil  : Running shell command: git rev-parse HEAD
2023-11-20T20:44:56.344Z  INFO 5 --- [ctor-http-nio-4] i.p.s.c.mirrorservice.mirrors.ShellUtil  : e76dde7188a2d6f927f9ea8d91efdd8cc7999461

In conclusion, mirror service will maintain one local mirror per remote repo. If many config-servers are created pointing to the same remote repo, then they will share the same internal mirror within mirror service. To make sure the local mirror is in sync with the remote repo, refreshing the mirror is encouraged. There are 5 methods for refreshing a mirror.

1 - cf config-server-sync-mirrors my-config-server via the cf CLI plugin
2 - Manual sync via dashboard
3 - cf update-service my-config-server -c ‘{“update-git-repos”: true }’
4 - Adding the parameter periodic=true to the config-server configuration. This feature is available from SCS v3.1.2+
5 - Hitting the config-server's actuator endpoint:

curl -X POST -H "Authorization: $(cf oauth-token)" -H "Content-Type: application/json" https://config-server-a5782192-8036-4f57-8312-4756a2604240.apps.example.com/actuator/refreshmirrors

Additional Information

'git clone --mirror' - This fetches all branches/tags and creates an exact replica of the original repository.