Rally - On-premises: Resolving VM/Docker swarm IP mismatch
search cancel

Rally - On-premises: Resolving VM/Docker swarm IP mismatch


Article ID: 139821


Updated On:


Rally On-Premise


When the IP address of a VM has changed, sometimes the Docker swarm IP address does not update to match.  This can cause some problems with Docker swarm certificate rotation and thereby a failure of the Docker swarm to start up.  The swarm IP should match the IP address that is assigned to the VM.


The Docker swarm IP can be verified in two ways; via the Cluster link in the on-premise dashboard or command line using a docker command.


To verify the swarm IP via the dashboard

Log into the dashboard using the following address: https://<IP_Address>:8800/dashboard

Click Cluster in the top navigation menu.  The IP address in the Image column should match the IP assigned to the VM



To verify via the command line

SSH into the services appliance as the ops user

Run the following command:

$ docker service ls


Sample output:

ID                  NAME                                       MODE                REPLICAS            IMAGE                                                                 PORTS
mfzomta5lgf4        premkit_replicated                         replicated          1/1                 registry.replicated.com/library/premkit:1.3.0                         
j78klqas9i5o        repl10aeecc4_admin                         replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-admin:1.0.21-2.0.1              
517t75x14vqa        repl10aeecc4_alm                           replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1                 *:30000->7001/tcp
w2x9harbzxsv        repl10aeecc4_analytics                     replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1                 *:30001->8080/tcp
kyloao79jb51        repl10aeecc4_burro                         replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-burro:1.3906.0-2.0            
lzhq93czac58        repl10aeecc4_licensing                     replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-licensing:1.1.5-2.0            
n5g5mrgp5sz5        repl10aeecc4_proxy                         replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-proxy:1.3.5-2.0                 *:80->80/tcp,*:443->443/tcp
ppb9e64k0e9e        repl10aeecc4_smtp                          replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-smtp:1.0.0-2.0                
22wlod0g9sb9        repl10aeecc4_solr                          replicated          1/1                 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1                 *:30002->8081/tcp
ad8v70iecnhv        replicated_replicated                      replicated          1/1                 quay.io/replicated/replicated:stable-2.32.1                           *:9874->9874/tcp,*:9878->9878/tcp
p2v5k12gv1l0        replicated_replicated-operator             global              0/1                 <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1      
sodfcg8anim6        replicated_replicated-ui                   replicated          1/1                 quay.io/replicated/replicated-ui:stable-2.32.1                        *:8800->8800/tcp
vodc0oaols6a        retraced_retraced-api                      replicated          1/1                 registry.replicated.com/library/retraced:1.3.13                       
ub8j27611yde        retraced_retraced-api-bootstrapper         replicated          0/1                 registry.replicated.com/library/retraced:1.3.13                       
khkrcsl8ehy4        retraced_retraced-cron                     replicated          1/1                 registry.replicated.com/library/retraced:1.3.13                       
2xyfkyyynjdc        retraced_retraced-display-template-setup   replicated          0/1                 registry.replicated.com/library/retraced:1.3.13                       
fs7vfe7n3pyr        retraced_retraced-nsqd                     replicated          1/1                 registry.replicated.com/library/retraced-nsq:v1.0.0-compat-20190117   
cdq8e1o9ylv6        retraced_retraced-pg-migrate               replicated          0/1                 registry.replicated.com/library/retraced:1.3.13                       
q5ubw40nwy5h        retraced_retraced-postgres                 replicated          1/1                 registry.replicated.com/library/retraced-postgres:10.6-20190125       
ny9vwndaddtu        retraced_retraced-processor                replicated          1/1                 registry.replicated.com/library/retraced:1.3.13                       
mqq2v4dys6sg        statsd_replicated                          replicated          1/1                 registry.replicated.com/library/statsd-graphite:1.0.1                


The IP addresses highlighted above should match the IP address assigned to the VM



Release : 2.0, 2.01



If the swarm IP addresses do not match the VM's IP address, there are a couple of solutions

There are bootup scripts that try to align the swarm IP address with the VM IP address.  Because of this, a reboot should first be attempted, particularly if the IP address was just changed.

If a reboot does not resolve this issue, then a command can be manually run to update

$ docker service update replicated_replicated-operator --image "<New_IP_Address>:9874/replicated/replicated-operator:stable-2.32.1"



$ docker service update replicated_replicated-operator --image "<HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1"


Sample output:

image <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1 could not be accessed on a registry to record
its digest. Each node will access <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1 independently,
possibly leading to different nodes running different
versions of the image.

overall progress: 0 out of 1 tasks
isn18szx8ir5: No such image: <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.3…