Rally on-premise showing "Loading" screen accessing the dashboard

Products

• Rally On-Premise

Issue/Introduction

The Rally dashboard is inaccessible and only showing a "Loading" prompt.

For new installs this issue will prevent the appliance from being installed entirely. For existing appliances installed prior to 11/13/2022, this will not impact the Rally application itself but will prevent the customer from being able to access the dashboard for any and all administrative tasks such as running snapshots, updating system configurations, stopping/starting the app, etc.

 

Environment

Release : 2.0+

Cause

This issue can occur when the Premkit container certificate expired on November 13, 2022.

Resolution

1. Enable root for these steps

2. Run docker ls, expect replicated_replicated and replicated_replicated-operator will be down (0/1)

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
mdockernamembq replicated_replicated replicated 0/1 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
wdockernamemd8 replicated_replicated-operator global 0/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
zdockernamempz replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
pdockernamemqu retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
xdockernamem0g retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
mdockernamem9c retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
4dockernamemyu retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
ldockernamemt1 retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
wdockernamemfg retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
odockernamemnq retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
4dockernamemi9 retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39

 

3. Run docker service logs on replicated_replicated and verify premkit ssl cert has expired. The date should be the same or roughly close to 2022-11-13.

ERRO 2022-12-20T21:56:42+00:00 daemon/daemon.go:405 Retraced audit log service failed: ensure premkit started: swarm `premkit.Manager` failed to create Premkit TLS secrets: create key pair for premkit: get existing keypair: verify certificate: x509: certificate has expired or is not yet valid: current time 2022-12-20T21:56:42Z is after 2022-11-13T18:41:07Z

# docker service logs replicated_replicated
[email protected] | INFO 2022-12-20T21:56:33+00:00 daemon/daemon.go:62 Starting Replicated version 2.49.0 (git="6f6770c", date="2020-10-28 16:41:13 +0000 UTC")
[email protected] | INFO 2022-12-20T21:56:33+00:00 [pkg/docker] client.go:123 Docker client initialized for API version 1.40
[email protected] | INFO 2022-12-20T21:56:34+00:00 crypto/certs/manager.go:80 New key pair for TLS configuration generated
[email protected] | WARN 2022-12-20T21:56:34+00:00 [scheduler/swarm] config.go:49 Config.SwarmIngressAddress empty
[email protected] | INFO 2022-12-20T21:56:34+00:00 eventmanager/event_manager.go:30 No event listener registered for event "RELOAD_AGENT_CLIENT_CA"
[email protected] | INFO 2022-12-20T21:56:34+00:00 [scheduler/swarm] client.go:37 Docker client initialized for API version 1.40
[email protected] | INFO 2022-12-20T21:56:34+00:00 daemon/daemon.go:1045 Starting TCP API server...
[email protected] | ERRO 2022-12-20T21:56:35+00:00 [scheduler/swarm] manager.go:143 swarm `premkit.Manager` failed to create Premkit TLS secrets: create key pair for premkit: get existing keypair: verify certificate: x509: certificate has expired or is not yet valid: current time 2022-12-20T21:56:35Z is after 2022-11-13T18:41:07Z

 

4. Scale down replicated_replicated using docker service scale replicated_replicated=0

Verify replicated_replicated now shows (0/0)

# docker service scale replicated_replicated=0
replicated_replicated scaled to 0
overall progress: 0 out of 0 tasks
verify: Service converged

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
<Dockerid> replicated_replicated replicated 0/0 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
<Dockerid> replicated_replicated-operator global 0/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
<Dockerid> replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
<Dockerid> retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
<Dockerid> retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Dockerid> retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
<Dockerid> retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Dockerid> retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
<Dockerid> retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Dockerid> retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
<Dockerid> retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39

 

5. Run ls on secrets directory, expect certs from Nov 13 2020.

# ls -latr /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
-r--------. 1 replicated docker 1675 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.key
-r--r--r--. 1 replicated docker 1168 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.crt
-r--r-----. 1 replicated docker 1679 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-premkit_replicated.host.key
-r--r--r--. 1 replicated docker 1318 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-premkit_replicated.host.crt
-r--r-----. 1 replicated docker 1679 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-retraced_retraced-api.host.key
-r--r--r--. 1 replicated docker 1326 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-retraced_retraced-api.host.crt
-r--r-----. 1 replicated docker 1675 Dec 20 14:52 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.17.71.220.host.key
-r--r--r--. 1 replicated docker 1289 Dec 20 14:52 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.17.71.220.host.crt
-r--r-----. 1 replicated docker 1675 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.253.5.186.host.key
-r--r--r--. 1 replicated docker 1289 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.253.5.186.host.crt
-r--------. 1 replicated docker 1 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.crt.info

 

6. Remove the entire secrets dir. Then verify they are gone!

# rm -rf /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
# ls -latr /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
ls: cannot access /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*: No such file or directory

 

7. Scale replicated_replicated up with docker service scale replicated_replicated=1

# docker service scale replicated_replicated=1
replicated_replicated scaled to 1
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged

 

8. You can verify replicated_replicated started correctly, the new cert is created and the service is healthy by running docker logs -f on the replicated/replicated:stable-2.49.0 container ID. Run docker ps first to get the replicated/replicated:stable-2.49.0 container ID.

Note your ID will differ from the one shown. You'll likely need to ctrl+c to exit docker logs.

# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
<ContainerID> registry.replicated.com/library/retraced:1.3.39 "/src/replicated-aud…" 17 seconds ago Up 13 seconds 3000/tcp retraced_retraced-api.1.x75u6fzih23xqyl71hyysfn9o
<ContainerID> registry.replicated.com/library/premkit:v1.3.7 "/usr/bin/premkit da…" 20 seconds ago Up 19 seconds 80/tcp, 443/tcp, 2080/tcp, 2443/tcp premkit_replicated.1.p5evo217c0chht9hk6bpyabgk
<ContainerID> replicated/replicated:stable-2.49.0 "/usr/bin/entrypoint…" 24 seconds ago Up 23 seconds 9874/tcp, 9876-9877/tcp, 9879/tcp replicated_replicated.1.2bmcnq2qhs9d3w5wskhoad8eu
<ContainerID> replicated/replicated-ui:stable-2.49.0 "/usr/bin/replicated…" 6 minutes ago Up 6 minutes 8800/tcp replicated_replicated-ui.1.mm2gm7ujigrcvnm7n39fzn32h
<ContainerID> registry.replicated.com/library/retraced-postgres:1.3.39 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 5432/tcp retraced_retraced-postgres.1.oe5bjxy46dzentbc0ttzcuc0v
<ContainerID> registry.replicated.com/library/retraced:1.3.39 "/src/replicated-aud…" 8 minutes ago Up 8 minutes 3000/tcp retraced_retraced-processor.1.e3yzvxtyci8ka1mz9wnxenu3w
<ContainerID> registry.replicated.com/library/retraced-nsq:1.3.39 "/bin/sh -c nsqd" 8 minutes ago Up 8 minutes 4150-4151/tcp, 4160-4161/tcp, 4170-4171/tcp retraced_retraced-nsqd.1.s5q9xry5ao3pyjuimvp0wzsia
<ContainerID> registry.replicated.com/library/retraced:1.3.39 "/bin/sh -c '/usr/lo…" 8 minutes ago Up 8 minutes 3000/tcp retraced_retraced-cron.1.lzix1rxkwka4574st50pfvn38


[root@localhost ~]# docker logs -f <ContainerID>
INFO 2022-12-20T22:00:37+00:00 daemon/daemon.go:62 Starting Replicated version 2.49.0 (git="6f6770c", date="2020-10-28 16:41:13 +0000 UTC")
INFO 2022-12-20T22:00:37+00:00 [pkg/docker] client.go:123 Docker client initialized for API version 1.40
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:78 Creating new certificate authority
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:82 New certificate authority created successfully
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:120 Creating new default certificate for address=<IP_ADDR>
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:353 Successfully generated new keypair for address=<IP_ADDR>, cns=[<IP_ADDR>]
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:124 New default certificate created successfully
INFO 2022-12-20T22:00:38+00:00 crypto/certs/manager.go:80 New key pair for TLS configuration generated
WARN 2022-12-20T22:00:38+00:00 [scheduler/swarm] config.go:49 Config.SwarmIngressAddress empty
INFO 2022-12-20T22:00:38+00:00 eventmanager/event_manager.go:30 No event listener registered for event "RELOAD_AGENT_CLIENT_CA"
INFO 2022-12-20T22:00:38+00:00 [scheduler/swarm] client.go:37 Docker client initialized for API version 1.40
INFO 2022-12-20T22:00:38+00:00 daemon/daemon.go:1045 Starting TCP API server...
INFO 2022-12-20T22:00:39+00:00 crypto/ca.go:353 Successfully generated new keypair for address=0.0.0.0, cns=[premkit_replicated]
INFO 2022-12-20T22:00:40+00:00 [pkg/piper] cmd.go:174 Piper result failed for file /var/lib/snapshots/files/db.dump: lstat /host/var/lib/snapshots/files/db.dump: no such file or directory
INFO 2022-12-20T22:00:40+00:00 snapshots/server/handlers.go:580 Store does not have a database. Resetting local db.
INFO 2022-12-20T22:00:40+00:00 snapshots/server/server.go:149 Snapshot server started.
INFO 2022-12-20T22:00:42+00:00 premkit/controller.go:71 Premkit started successfully
INFO 2022-12-20T22:00:42+00:00 crypto/ca.go:353 Successfully generated new keypair for address=0.0.0.0, cns=[retraced_retraced-api]
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-cron (id: mybc1eau7e9cvliht6lktprm7)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-postgres (id: oy2n9k39kxnql8t4ioqwwtme7)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-pg-migrate (id: wl3cud8g8pfgmxzz1chmj63uo)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-nsqd (id: l7dinqc28st1t5ruaf8x6bjcu)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-api-bootstrapper (id: x1139o14xs0g28t5dxktklx87)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-display-template-setup (id: 41ieyj6y23yujn9psixd9ohwl)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-api (id: ptp3wuv4ekqu8b3mfy1emmf8n)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-processor (id: 4az7ie181qi9lhs9x1cutkmux)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] manager.go:110 Started premkit service retraced
INFO 2022-12-20T22:00:42+00:00 pkg/health/await_healthy.go:16 Waiting for service retraced
INFO 2022-12-20T22:00:42+00:00 pkg/health/await_healthy.go:21 Service retraced is ready

 

 

9. Next we need to retag the replicated_replicated-operator. (this step may actually not be needed for an existing appliance, it is still unclear but according to Replicated support it's probably ok to leave this alone and it will self-correct the next time the appliance is restarted). 

Also important to note your IP address will differ from what is below.

# docker tag replicated/replicated-operator:stable-2.49.0 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0

 

10. Finally verify replicated_replicated and replicated_replicated-operator are scaled up (1/1)

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
<Docker_ID> premkit_replicated replicated 1/1 registry.replicated.com/library/premkit:v1.3.7
<Docker_ID> replicated_replicated replicated 1/1 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
<Docker_ID> replicated_replicated-operator global 1/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
<Docker_ID> replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
<Docker_ID> retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
<Docker_ID> retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Docker_ID> retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
<Docker_ID> retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Docker_ID> retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
<Docker_ID> retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
<Docker_ID> retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
<Docker_ID> retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39

At this point, you should be able to access the Replicated dashboard on port 8800.