DX Platform 20.2.1 - Installation succeeded but unable login
search cancel

DX Platform 20.2.1 - Installation succeeded but unable login

book

Article ID: 220363

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

After successfully installed the DX Platform 20.2.1:

“[dxi-platform-init] [ OK ] Check log in xx.xx.xx.xx:/nfs/ca/dxi/dxi-platform-init-pod/dxi-platform-init-pod.log before login to DXI Manager

Installation succeeded

To create your first tenant, open this URL: http://apmservices-gateway.xx.xx.xx.xx.nip.io/dxiportal. Log in using following credentials: "masteradmin" tenant and "masteradmin" user. Use the password that you entered as a masteradmin password during the installation”

 

http://apmservices-gateway.xx.xx.xx.xx.nip.io/dxiportal is up and running, however the login doesn't work and  /nfs/ca/dxi/dxi-platform-init-pod/dxi-platform-init-pod.log is reporting the following:

[ *** RESULTS *** ]
1.reset.masteradmin.password.sh -> FAILED
2.create.ga.account.sh -> FAILED
3.apm.onboard.sh -> FAILED

 

- Below the errors we se in the logs:

[default@dxi-adminui-64d86f9c78-p6c28 logs]$ vi ca-ess.log
ready
ResourceCacheMgr:
Fallback hardcoded strings provided by: ess-message.properties

No localized strings read from database!?
TenantDataAndMappingCache:
Error: Tenant data not initialized!!

[CentralFactory,main] INFO   - TID[92648117] initialize, CentralFactory Successfully Initialized ........
[CacheRefreshCheck,main] INFO   - TID[92648117] initializeAndRunNow, CacheRefreshCheck Successfully initialized
[ESSInitServlet,main] INFO   - TID[92648117] ESSInitServlet initialized with out of box settings
[ESSInitServlet,main] INFO   - TID[92648117] ess webapp started successfully. Ready to serve REST requests...
[ESSInitServlet,main] INFO   - TID[92648117] Swagger customizations injected
[JmsMQRequester,essCacheRefreshCheck] INFO   - TID[92648117] MQ Broker URL from instance.properties set to failover:(tcp://axaservices-amq:61616)?randomize=false&priorityBackup=true
[JmsMQRequester,essCacheRefreshCheck] INFO   - TID[92648117] JmsMQRequester successfully reinitialized connection to failover:(tcp://axaservices-amq:61616)?randomize=false&priorityBackup=true
[JmsMQRequester,essCacheRefreshCheck] INFO   - TID[92648117] ConfigEntrysGetRequest sent with CorrelationID=ESS_instance_92648117.1-1626773222851
[EMMCacheRefresher,essCacheRefreshCheck] ERROR  - TID[92648117] 5271596: JMS Request processing timeout, please wait for background processing
com.ca.emm.corejsvr.ExceptionWithNC: JMS Request processing timeout, please wait for background processing
        at com.ca.emm.svcsenv.svrcommon.jmsclient.JmsMQRequester.makeRequest(JmsMQRequester.java:381)
        at com.ca.emm.svcsenv.svrcommon.jmsclient.JmsMQRequester.makeRequest(JmsMQRequester.java:423)
        at com.ca.emm.svcsenv.svrcommon.dao.SystemConfigViaJMS.getConfigsLastUpdated(SystemConfigViaJMS.java:43)
        at com.ca.emm.svcsenv.svrcommon.EMMCacheRefresher.doRefresh(EMMCacheRefresher.java:194)
        at com.ca.emm.svcsenv.svrcommon.EMMCacheRefresher.run(EMMCacheRefresher.java:134)
        at com.ca.emm.corejsvr.cache.CacheRefreshCheck$CallbackRunner.run(CacheRefreshCheck.java:272)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[EMMCacheRefresher,essCacheRefreshCheck] ERROR  - TID[92648117] 3011599: Internal Error: Unable to complete this cache refresh cycle: 5271596: JMS Request processing timeout, please wait for background processing
[JmsMQRequester,essCacheRefreshCheck] INFO   - TID[92648117] ConfigEntrysGetRequest sent with CorrelationID=ESS_instance_92648117.1-1626773222852
[EMMCacheRefresher,essCacheRefreshCheck] ERROR  - TID[92648117] 5271596: JMS Request processing timeout, please wait for background processing
com.ca.emm.corejsvr.ExceptionWithNC: JMS Request processing timeout, please wait for background processing
        at com.ca.emm.svcsenv.svrcommon.jmsclient.JmsMQRequester.makeRequest(JmsMQRequester.java:381)
        at com.ca.emm.svcsenv.svrcommon.jmsclient.JmsMQRequester.makeRequest(JmsMQRequester.java:423)
        at com.ca.emm.svcsenv.svrcommon.dao.SystemConfigViaJMS.getConfigsLastUpdated(SystemConfigViaJMS.java:43)
        at com.ca.emm.svcsenv.svrcommon.EMMCacheRefresher.doRefresh(EMMCacheRefresher.java:194)
        at com.ca.emm.svcsenv.svrcommon.EMMCacheRefresher.run(EMMCacheRefresher.java:134)
        at com.ca.emm.corejsvr.cache.CacheRefreshCheck$CallbackRunner.run(CacheRefreshCheck.java:272)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[EMMCacheRefresher,essCacheRefreshCheck] ERROR  - TID[92648117] 3011599: Internal Error: Unable to complete this cache refresh cycle: 5271596: JMS Request processing timeout, please wait for background processing

 

[root@test meta.import]# kubectl logs -f dxi-platform-init-pod -n dxi
[ RUNNING 1.reset.masteradmin.password.sh]

Waiting for adminui to be up...............................................................................................................................................................................................................................................................................................................
Resetting password for masteradmin [Cohort : masteradmin]

Obtaining security token from ESS...
STATUS:..........................................................................................
Failed to obtain authorization token... Exiting.

[ END 1.reset.masteradmin.password.sh ]


[ RUNNING 2.create.ga.account.sh]
Waiting for adminui to be up...............................................................................................................................................................................................................................................................................................................

Creating Global Administrator: NOTIFY-FILTER-GA

Obtaining security token from ESS...
STATUS:............................................................ERROR! Failed to obtain authorization token... Exiting.

[ END 2.create.ga.account.sh ]


[ RUNNING 3.apm.onboard.sh]
Waiting for jarvis-apis to be up...
template ao_apm_tt_analyzer loaded into ES.

Error! Failed to load doc type - apm_tt.
{"_message":"Mapping with similar doct_type_id and doc_type_version exists.","_code":"1011"}
HTTPStatus=409

Error! Failed to load doc type - apm_itoa_alarms_apm_1.
{"_message":"Mapping with similar doct_type_id and doc_type_version exists.","_code":"1011"}
HTTPStatus=409

Error! Failed to load doc type - itoa_inventory_apm.
{"_message":"Mapping with similar doct_type_id and doc_type_version exists.","_code":"1011"}
HTTPStatus=409

[ END 3.apm.onboard.sh ]


[ *** RESULTS *** ]
3.apm.onboard.sh -> FAILED
1.reset.masteradmin.password.sh -> FAILED
2.create.ga.account.sh -> FAILED

Environment

 DX Platform 20.2.1 on Kubernetes cluster

Resolution

We have resolved the issue by running below scripts from the postgres pod as it seems they didn't get executed properly during the installation:

a.- Get into the postgres pod

b.- Navigate to /opt/app-root/src/postgresql-init directory and run below three scripts:
13_dxiconfigure_securestore.sh
14_dxiconfigure_dbpopulator_templates.sh
15_dxiconfigure_dbpopulator_serviceproviders.sh

eg:

dxi-postgresql-844bb96d4d-k7vws                        1/1     Running     0          32m

[root@test /]# kubectl exec -it dxi-postgresql-844bb96d4d-k7vws bash -n dxi
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-4.2$ dir
axa_server_java_libs  migration  notify-filter-resources  postgresql-init  postgresql-start  resources  spp-resources
bash-4.2$ cd postgresql-init/
bash-4.2$ dir
00_create_aoplatform_db.sh        04_create_apmpe_user_db.sh  08_config_doi_db.sh            12_config_grafana.sh
01_create_mcc_maa_doi_schemas.sh  05_config_spp.sh            09_create_dspintegrator_db.sh  13_dxiconfigure_securestore.sh
02_create_dsp_user_db_tables.sh   06_config_notify_filter.sh  10_config_automic.sh           14_dxiconfigure_dbpopulator_templates.sh
03_create_cpa_db.sh               07_create_doi_db.sh         11_create_grafana_db.sh        15_dxiconfigure_dbpopulator_serviceproviders.sh
bash-4.2$ ./13_dxiconfigure_securestore.sh
+ echo 'CA_EMM_HOME: /opt/ca/aoplatform'
CA_EMM_HOME: /opt/ca/aoplatform
...

c.- Scale down and up the adminui, readsever and amp pods in the following order:
[root@test /]# kubectl scale --replicas=0 deployment -n dxi dxi-adminui
deployment.apps/dxi-adminui scaled
[root@test /]# kubectl scale --replicas=0 deployment -n dxi dxi-readserver
deployment.apps/dxi-readserver scaled
[root@test /]# kubectl scale --replicas=0 deployment -n dxi axaservices-amq
deployment.apps/axaservices-amq scaled
[root@test /]# kubectl scale --replicas=1 deployment -n dxi axaservices-amq
deployment.apps/axaservices-amq scaled
[root@test /]# kubectl scale --replicas=0 deployment -n dxi dxi-readserver
deployment.apps/dxi-readserver scaled
[root@test /]# kubectl scale --replicas=1 deployment -n dxi dxi-readserver
deployment.apps/dxi-readserver scaled
[root@test /]# kubectl scale --replicas=1 deployment -n dxi dxi-adminui
deployment.apps/dxi-adminui scaled


d.- Open the browser and go to the DX Platfrom and try to login with default user/pass:
http://apmservices-gateway.xx.xx.xx.xx.nip.io/dxiportal/#/dxiportal/login

User: masteradmin

Pass: masteradmin1234!

 

If the problem persists, please contact Broadcom Support.

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815

Powered by Wolken Software