How does SCS Config Server interact with Config Client and Mirror Service

Products

• VMware Tanzu Application Service

Issue/Introduction

Here is the descriptions of Config Server architecture in Spring Cloud Services for VMware Tanzu documentation.

The Config Server serves properties from multiple sources, including one or more Git repositories. For Git repositories, the Config Server makes use of an internal "mirror service" to create a mirror of each Git repository used. These mirrors are local to the VMware Tanzu Application Service for VMs (TAS for VMs) foundation, and the Config Server serves properties to client applications from these mirrors, rather than from the external Git repositories themselves.

This KB article attempts to provide more information about how the Config Server interacts with Config Client and Mirror Service.

Environment

Spring Cloud Services for VMware Tanzu v3.2+

Resolution

There are several configuration parameters which could affect the behaviour of the Config Client, Config Server and Mirror Service as described in the document (also shown in below image).

The Config Server service instance is actually backed by application running on TAS. When the Config Server app starts, it will not connect to either Config Client app or the Mirror Service. Instead, it will be empty in its local cache and waiting for request from the Config Client app.

When the Config Client app starts, it will attempt to load configuration properties from the binding Config Server service instance, which would consequently trigger a "git clone" operation to fetch data from the Mirror Service.

As it's mentioned in the document, you can manually refresh a Config Server service instance's Git mirrors using Apps Manager, the Cloud Foundry Command Line Interface ( the cf config-server-sync-mirrors command added by the Spring Cloud Services plug-in for the cf CLI), or a custom Spring Boot Actuator endpoint on the service instance's backing app. All those 3 approaches will do the same thing. Take the Spring Boot Actuator endpoint /actuator/refreshmirrors as an example, when the endpoint is called, it will trigger a sync operation between the Mirror Service and the external Git repo. If the Config Server has no local cached data at this time, the Config Server will also clone data from Mirror Service. But if Config Server already has data in its local cache, the clone operation would only be triggered when

• refreshRate parameter is set to 0 (default value)
• or the previous sync operation is done refreshRate seconds ago

Another way is to the cf update-service command to refresh Git mirrors for a service instance by passing an update-git-repos parameter to the command. For example,

cf update-service SERVICE_NAME -c '{ "update-git-repos": true }'

The above command actually executes a git fetch origin in the mirror service, which does not depend on any branch/tag or commit configuration. Each Config Server instance has a local clone of the mirrored repo in its backing app container. This local clone is fetched from mirror service using the label (branch/tag/commit-sha). Therefore, the update-git-repos method does not change anything in the Config Server until the client application bound to the service instance requests the configuration again (in case of restart). Upon receiving the request, the Config Server then fetches the updated repo from the mirror service and provides the updated configuration in the specific label to the client.

The Config Server's Git mirrors could also be refreshed automatically and periodically if periodic is set to true for the service instance. The Mirror Service has a scheduled job which will be run every 5 minutes. In each 5m period the job will try to find all mirrors with periodic=true and run git fetch for each one. This only happens between Mirror Service and remote Git repo. The Config Server itself won't be involved. 

If periodic is set to false for the Config Server service instance, then it will only sync with the mirror service through manual operation (CLI, Dashboard or Actuator endpoint). And the following activities will not cause the mirror service to sync with remote Git repo either.

• New client app connecting to the Config Server instance and fetching properties
• Config Server crash and restart
• Config Server restage
• Config Server update through either upgrade-all-service-instances errand or cf update-service CLI command

There will be a single mirror of each Git/Bitbucket repository, even if there are multiple Config Server instances configured with the same repo. And the mirror is created based on the configuration of the first Config Server instance. The rest of them will simply use the mirror without overriding the configuration. For example, Config Sever A and B are created with the same Git/Bitbucket repository (maybe different branches/labels). Config Server A is created first with periodic=false while  B is created later with periodic=true. Then both A and B will not have periodic refreshing of their mirror with remote Git/Bitbucket repo.

According to note in the document, timeout controls number of seconds that the Config Server will wait to acquire a connection to the mirrored repository and default value is 5. You might need to consider setting this parameter to a higher value (e.g. 30) when there are many Config Server service instances in a TAS environment, which means the Mirror Service could get overloaded under some situation (e.g. updating many diego_cell instances simutaneously during TAS/Isolation segment tile upgrade). timeout parameter could be set at git and git.repos object level as illustrated below.

{
    "count": 3,
    "git": {
        "uri": ####
        "timeout": ####
        "repos": {
            "mirrorservice-1": {
                "label": "master"
                "timeout": ####
            }
        }
    }

......
}

 

The git.timeout value will be overridden by git.repos.timeout value if it's set. if git.repos.timeout is not set, git.timeout would apply.

Each Config Server service instance has a backing app called config-server. The backing app is bound to a mirror service instance called mirror-svc. The name of the backing app (config-server) and the backing service instance (mirror-svc) is always the same for any Config Server service instance.

The mirror directory created in /var/vcap/store location of SCS broker VM is named as the MD5 digest of the repo URI (as illustrated below). So the mirror directory name will always be same for same Git/Bitbucket repositories.

spring-cloud-services/cbdccf80-####-####-a53d-cad26c8bba96:~# ls -l /var/vcap/store/mirror/
total 4
drwxr-xr-x 3 mirror vcap 4096 Dec 30 04:46 54e015c####77f08c22f0c472ec

 

After cf update-service CLI command is executed on a Config Server service instance, the backing app config-server will restart with new configuration. Since the client apps will only load its properties from the Config Server during the startup, restarting the client apps is needed if changes would affect client app. For example, if user changes  "periodic","timeout","refreshRate", these have no impact on the properties read in by the Config Server, as the Git repo URI and label etc remain the same, then restart of the client app is not required. 

There is no guideline or metric to determine the number of applications which could be bound to a single Config Server instance. Normally, those applications only send one or two requests to the Config Server instance during startup. So N applications are bound to one Config Server instance, they will send about 2 * N requests to the Config Server instance which shouldn't be a problem for the Config Server instance. The only issue would be when all those applications start or restart at the same time. So the key point is to manage the number of concurrent re/start of those applications.