search cancel

Rally on-premise showing "Loading" screen accessing the dashboard

book

Article ID: 257204

calendar_today

Updated On:

Products

CA Agile Central On Premise (Rally)

Issue/Introduction

The Rally dashboard is inaccessible and only showing a "Loading" prompt.

For new installs this issue will prevent the appliance from being installed entirely. For existing appliances installed prior to 11/13/2022, this will not impact the Rally application itself but will prevent the customer from being able to access the dashboard for any and all administrative tasks such as running snapshots, updating system configurations, stopping/starting the app, etc.

 

Environment

Release : 2.0+

Cause

This issue can occur when the Premkit container certificate expired on November 13, 2022.

Resolution

1. Enable root for these steps

2. Run docker ls, expect replicated_replicated and replicated_replicated-operator will be down (0/1)

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
mz0vmb9i54bq replicated_replicated replicated 0/1 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
wpfyi0v1xsd8 replicated_replicated-operator global 0/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
z6qhqam7vhpz replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
ptp3wuv4ekqu retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
x1139o14xs0g retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
mybc1eau7e9c retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
41ieyj6y23yu retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
l7dinqc28st1 retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
wl3cud8g8pfg retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
oy2n9k39kxnq retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
4az7ie181qi9 retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39

 

3. Run docker service logs on replicated_replicated and verify premkit ssl cert has expired. The date should be the same or roughly close to 2022-11-13.

ERRO 2022-12-20T21:56:42+00:00 daemon/daemon.go:405 Retraced audit log service failed: ensure premkit started: swarm `premkit.Manager` failed to create Premkit TLS secrets: create key pair for premkit: get existing keypair: verify certificate: x509: certificate has expired or is not yet valid: current time 2022-12-20T21:56:42Z is after 2022-11-13T18:41:07Z

# docker service logs replicated_replicated
[email protected] | INFO 2022-12-20T21:56:33+00:00 daemon/daemon.go:62 Starting Replicated version 2.49.0 (git="6f6770c", date="2020-10-28 16:41:13 +0000 UTC")
[email protected] | INFO 2022-12-20T21:56:33+00:00 [pkg/docker] client.go:123 Docker client initialized for API version 1.40
[email protected] | INFO 2022-12-20T21:56:34+00:00 crypto/certs/manager.go:80 New key pair for TLS configuration generated
[email protected] | WARN 2022-12-20T21:56:34+00:00 [scheduler/swarm] config.go:49 Config.SwarmIngressAddress empty
[email protected] | INFO 2022-12-20T21:56:34+00:00 eventmanager/event_manager.go:30 No event listener registered for event "RELOAD_AGENT_CLIENT_CA"
[email protected] | INFO 2022-12-20T21:56:34+00:00 [scheduler/swarm] client.go:37 Docker client initialized for API version 1.40
[email protected] | INFO 2022-12-20T21:56:34+00:00 daemon/daemon.go:1045 Starting TCP API server...
[email protected] | ERRO 2022-12-20T21:56:35+00:00 [scheduler/swarm] manager.go:143 swarm `premkit.Manager` failed to create Premkit TLS secrets: create key pair for premkit: get existing keypair: verify certificate: x509: certificate has expired or is not yet valid: current time 2022-12-20T21:56:35Z is after 2022-11-13T18:41:07Z

 


4. Scale down replicated_replicated using docker service scale replicated_replicated=0

Verify replicated_replicated now shows (0/0)

# docker service scale replicated_replicated=0
replicated_replicated scaled to 0
overall progress: 0 out of 0 tasks
verify: Service converged

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
mz0vmb9i54bq replicated_replicated replicated 0/0 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
wpfyi0v1xsd8 replicated_replicated-operator global 0/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
z6qhqam7vhpz replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
ptp3wuv4ekqu retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
x1139o14xs0g retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
mybc1eau7e9c retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
41ieyj6y23yu retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
l7dinqc28st1 retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
wl3cud8g8pfg retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
oy2n9k39kxnq retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
4az7ie181qi9 retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39

 


5. Run ls on secrets directory, expect certs from Nov 13 2020.

# ls -latr /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
-r--------. 1 replicated docker 1675 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.key
-r--r--r--. 1 replicated docker 1168 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.crt
-r--r-----. 1 replicated docker 1679 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-premkit_replicated.host.key
-r--r--r--. 1 replicated docker 1318 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-premkit_replicated.host.crt
-r--r-----. 1 replicated docker 1679 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-retraced_retraced-api.host.key
-r--r--r--. 1 replicated docker 1326 Nov 13 2020 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-0.0.0.0-retraced_retraced-api.host.crt
-r--r-----. 1 replicated docker 1675 Dec 20 14:52 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.17.71.220.host.key
-r--r--r--. 1 replicated docker 1289 Dec 20 14:52 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.17.71.220.host.crt
-r--r-----. 1 replicated docker 1675 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.253.5.186.host.key
-r--r--r--. 1 replicated docker 1289 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/pkix-10.253.5.186.host.crt
-r--------. 1 replicated docker 1 Dec 20 14:54 /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/ca.crt.info

 


6. Remove the entire secrets dir. Then verify they are gone!

# rm -rf /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
# ls -latr /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*
ls: cannot access /var/lib/docker/volumes/replicated_replicated-data-volume/_data/secrets/*: No such file or directory

 


7. Scale replicated_replicated up with docker service scale replicated_replicated=1

# docker service scale replicated_replicated=1
replicated_replicated scaled to 1
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged

 


8. You can verify replicated_replicated started correctly, the new cert is created and the service is healthy by running docker logs -f on the replicated/replicated:stable-2.49.0 container ID. Run docker ps first to get the replicated/replicated:stable-2.49.0 container ID.

Note your ID will differ from the one shown. You'll likely need to ctrl+c to exit docker logs.

# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
be63f2b15454 registry.replicated.com/library/retraced:1.3.39 "/src/replicated-aud…" 17 seconds ago Up 13 seconds 3000/tcp retraced_retraced-api.1.x75u6fzih23xqyl71hyysfn9o
a07ced785456 registry.replicated.com/library/premkit:v1.3.7 "/usr/bin/premkit da…" 20 seconds ago Up 19 seconds 80/tcp, 443/tcp, 2080/tcp, 2443/tcp premkit_replicated.1.p5evo217c0chht9hk6bpyabgk
0d0022f448dc replicated/replicated:stable-2.49.0 "/usr/bin/entrypoint…" 24 seconds ago Up 23 seconds 9874/tcp, 9876-9877/tcp, 9879/tcp replicated_replicated.1.2bmcnq2qhs9d3w5wskhoad8eu
9b5ddd96da77 replicated/replicated-ui:stable-2.49.0 "/usr/bin/replicated…" 6 minutes ago Up 6 minutes 8800/tcp replicated_replicated-ui.1.mm2gm7ujigrcvnm7n39fzn32h
ef0a3e88d70e registry.replicated.com/library/retraced-postgres:1.3.39 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 5432/tcp retraced_retraced-postgres.1.oe5bjxy46dzentbc0ttzcuc0v
0f233c64063f registry.replicated.com/library/retraced:1.3.39 "/src/replicated-aud…" 8 minutes ago Up 8 minutes 3000/tcp retraced_retraced-processor.1.e3yzvxtyci8ka1mz9wnxenu3w
05ce8bd65aea registry.replicated.com/library/retraced-nsq:1.3.39 "/bin/sh -c nsqd" 8 minutes ago Up 8 minutes 4150-4151/tcp, 4160-4161/tcp, 4170-4171/tcp retraced_retraced-nsqd.1.s5q9xry5ao3pyjuimvp0wzsia
5f9d811a2af0 registry.replicated.com/library/retraced:1.3.39 "/bin/sh -c '/usr/lo…" 8 minutes ago Up 8 minutes 3000/tcp retraced_retraced-cron.1.lzix1rxkwka4574st50pfvn38


[[email protected] ~]# docker logs -f 0d0022f448dc
INFO 2022-12-20T22:00:37+00:00 daemon/daemon.go:62 Starting Replicated version 2.49.0 (git="6f6770c", date="2020-10-28 16:41:13 +0000 UTC")
INFO 2022-12-20T22:00:37+00:00 [pkg/docker] client.go:123 Docker client initialized for API version 1.40
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:78 Creating new certificate authority
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:82 New certificate authority created successfully
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:120 Creating new default certificate for address=10.253.5.186
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:353 Successfully generated new keypair for address=10.253.5.186, cns=[10.253.5.186]
INFO 2022-12-20T22:00:38+00:00 crypto/ca.go:124 New default certificate created successfully
INFO 2022-12-20T22:00:38+00:00 crypto/certs/manager.go:80 New key pair for TLS configuration generated
WARN 2022-12-20T22:00:38+00:00 [scheduler/swarm] config.go:49 Config.SwarmIngressAddress empty
INFO 2022-12-20T22:00:38+00:00 eventmanager/event_manager.go:30 No event listener registered for event "RELOAD_AGENT_CLIENT_CA"
INFO 2022-12-20T22:00:38+00:00 [scheduler/swarm] client.go:37 Docker client initialized for API version 1.40
INFO 2022-12-20T22:00:38+00:00 daemon/daemon.go:1045 Starting TCP API server...
INFO 2022-12-20T22:00:39+00:00 crypto/ca.go:353 Successfully generated new keypair for address=0.0.0.0, cns=[premkit_replicated]
INFO 2022-12-20T22:00:40+00:00 [pkg/piper] cmd.go:174 Piper result failed for file /var/lib/snapshots/files/db.dump: lstat /host/var/lib/snapshots/files/db.dump: no such file or directory
INFO 2022-12-20T22:00:40+00:00 snapshots/server/handlers.go:580 Store does not have a database. Resetting local db.
INFO 2022-12-20T22:00:40+00:00 snapshots/server/server.go:149 Snapshot server started.
INFO 2022-12-20T22:00:42+00:00 premkit/controller.go:71 Premkit started successfully
INFO 2022-12-20T22:00:42+00:00 crypto/ca.go:353 Successfully generated new keypair for address=0.0.0.0, cns=[retraced_retraced-api]
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-cron (id: mybc1eau7e9cvliht6lktprm7)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-postgres (id: oy2n9k39kxnql8t4ioqwwtme7)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-pg-migrate (id: wl3cud8g8pfgmxzz1chmj63uo)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-nsqd (id: l7dinqc28st1t5ruaf8x6bjcu)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-api-bootstrapper (id: x1139o14xs0g28t5dxktklx87)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-display-template-setup (id: 41ieyj6y23yujn9psixd9ohwl)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-api (id: ptp3wuv4ekqu8b3mfy1emmf8n)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] deploy.go:222 Updating service retraced_retraced-processor (id: 4az7ie181qi9lhs9x1cutkmux)
INFO 2022-12-20T22:00:42+00:00 [scheduler/swarm] manager.go:110 Started premkit service retraced
INFO 2022-12-20T22:00:42+00:00 pkg/health/await_healthy.go:16 Waiting for service retraced
INFO 2022-12-20T22:00:42+00:00 pkg/health/await_healthy.go:21 Service retraced is ready

 

 

9. Next we need to retag the replicated_replicated-operator. (this step may actually not be needed for an existing appliance, it is still unclear but according to Replicated support it's probably ok to leave this alone and it will self-correct the next time the appliance is restarted). 


Also important to note your IP address will differ from what is below.

# docker tag replicated/replicated-operator:stable-2.49.0 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0

 


10. Finally verify replicated_replicated and replicated_replicated-operator are scaled up (1/1)

# docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
jo2e47jl74o8 premkit_replicated replicated 1/1 registry.replicated.com/library/premkit:v1.3.7
mz0vmb9i54bq replicated_replicated replicated 1/1 replicated/replicated:stable-2.49.0 *:9874->9874/tcp, *:9878->9878/tcp
wpfyi0v1xsd8 replicated_replicated-operator global 1/1 10.17.71.220:9874/replicated/replicated-operator:stable-2.49.0
z6qhqam7vhpz replicated_replicated-ui replicated 1/1 replicated/replicated-ui:stable-2.49.0 *:8800->8800/tcp
ptp3wuv4ekqu retraced_retraced-api replicated 1/1 registry.replicated.com/library/retraced:1.3.39
x1139o14xs0g retraced_retraced-api-bootstrapper replicated 0/1 registry.replicated.com/library/retraced:1.3.39
mybc1eau7e9c retraced_retraced-cron replicated 1/1 registry.replicated.com/library/retraced:1.3.39
41ieyj6y23yu retraced_retraced-display-template-setup replicated 0/1 registry.replicated.com/library/retraced:1.3.39
l7dinqc28st1 retraced_retraced-nsqd replicated 1/1 registry.replicated.com/library/retraced-nsq:1.3.39
wl3cud8g8pfg retraced_retraced-pg-migrate replicated 0/1 registry.replicated.com/library/retraced:1.3.39
oy2n9k39kxnq retraced_retraced-postgres replicated 1/1 registry.replicated.com/library/retraced-postgres:1.3.39
4az7ie181qi9 retraced_retraced-processor replicated 1/1 registry.replicated.com/library/retraced:1.3.39


At this point, you should be able to access the Replicated dashboard on port 8800.