Rally - On-premises: Resolving VM/Docker swarm IP mismatch
Article ID: 139821
Updated On:
Products
Issue/Introduction
When the IP address of a VM has changed, sometimes the Docker swarm IP address does not update to match. This can cause some problems with Docker swarm certificate rotation and thereby a failure of the Docker swarm to start up. The swarm IP should match the IP address that is assigned to the VM.
The Docker swarm IP can be verified in two ways; via the Cluster link in the on-premise dashboard or command line using a docker command.
To verify the swarm IP via the dashboard
Log into the dashboard using the following address: https://<IP_Address>:8800/dashboard
Click Cluster in the top navigation menu. The IP address in the Image column should match the IP assigned to the VM
To verify via the command line
SSH into the services appliance as the ops user
Run the following command:
$ docker service ls
Sample output:
ID NAME MODE REPLICAS IMAGE PORTS
mfzomta5lgf4 premkit_replicated replicated 1/1 registry.replicated.com/library/premkit:1.3.0
j78klqas9i5o repl10aeecc4_admin replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-admin:1.0.21-2.0.1
517t75x14vqa repl10aeecc4_alm replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1 *:30000->7001/tcp
w2x9harbzxsv repl10aeecc4_analytics replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1 *:30001->8080/tcp
kyloao79jb51 repl10aeecc4_burro replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-burro:1.3906.0-2.0
lzhq93czac58 repl10aeecc4_licensing replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-licensing:1.1.5-2.0
n5g5mrgp5sz5 repl10aeecc4_proxy replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-proxy:1.3.5-2.0 *:80->80/tcp,*:443->443/tcp
ppb9e64k0e9e repl10aeecc4_smtp replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-smtp:1.0.0-2.0
22wlod0g9sb9 repl10aeecc4_solr replicated 1/1 <HOST_IP_ADDRESS>:9874/rallysoftware-ac2go-alm:23677-2.0.1 *:30002->8081/tcp
ad8v70iecnhv replicated_replicated replicated 1/1 quay.io/replicated/replicated:stable-2.32.1 *:9874->9874/tcp,*:9878->9878/tcp
p2v5k12gv1l0 replicated_replicated-operator global 0/1 <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1
sodfcg8anim6 replicated_replicated-ui replicated 1/1 quay.io/replicated/replicated-ui:stable-2.32.1 *:8800->8800/tcp
vodc0oaols6a retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.13
ub8j27611yde retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.13
khkrcsl8ehy4 retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.13
2xyfkyyynjdc retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.13
fs7vfe7n3pyr retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:v1.0.0-compat-20190117
cdq8e1o9ylv6 retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.13
q5ubw40nwy5h retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:10.6-20190125
ny9vwndaddtu retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.13
mqq2v4dys6sg statsd_replicated replicated 1/1 registry.replicated.com/library/statsd-graphite:1.0.1
The IP addresses highlighted above should match the IP address assigned to the VM
Environment
Release : 2.0, 2.01
Component : AGILE CENTRAL ON PREMISES
Resolution
If the swarm IP addresses do not match the VM's IP address, there are a couple of solutions
There are bootup scripts that try to align the swarm IP address with the VM IP address. Because of this, a reboot should first be attempted, particularly if the IP address was just changed.
If a reboot does not resolve this issue, then a command can be manually run to update
$ docker service update replicated_replicated-operator --image "<New_IP_Address>:9874/replicated/replicated-operator:stable-2.32.1"
Example:
$ docker service update replicated_replicated-operator --image "<HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1"
Sample output:
image <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1 could not be accessed on a registry to record
its digest. Each node will access <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.32.1 independently,
possibly leading to different nodes running different
versions of the image.
replicated_replicated-operator
overall progress: 0 out of 1 tasks
isn18szx8ir5: No such image: <HOST_IP_ADDRESS>:9874/replicated/replicated-operator:stable-2.3…
Feedback
