AEWS URL keeps looping with 401 unauthorized error

Products

• Autosys Workload Automation

Issue/Introduction

When they access AEWS URL,  they keep getting 401 unauthorized in a loop.   

waae_webservices_wrapper.log had some exceptions.

INFO   | jvm 1    | 2023/07/20 16:54:13 | 20-Jul-2023 16:54:13.588 INFO [WrapperStartStopAppMain] org.apache.catalina.startup.Catalina.start Server startup in [4,065] milliseconds
INFO   | jvm 1    | 2023/07/20 16:54:49 | Exception in thread "MetaDataLoader" java.lang.NullPointerException
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.waitMonitorStartup(AEConnMgr.java:117)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.executeQuery(AEConnMgr.java:106)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuery.execute(AEQuery.java:94)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuerySQL.execute(AEQuerySQL.java:40)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.metadata.MetaData.GetMetaProperties(MetaData.java:331)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.metadata.MetaData.<init>(MetaData.java:69)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.metadata.WsMetaData.<init>(WsMetaData.java:57)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.aehome.AEHomeInstance$MetaDataLoader.run(AEHomeInstance.java:144)
INFO   | jvm 1    | 2023/07/20 16:54:49 | 20-Jul-2023 16:54:49.727 WARNING [https-jsse-nio-9443-exec-7] org.apache.catalina.realm.JAASRealm.authenticate Login exception authenticating username [user123]
INFO   | jvm 1    | 2023/07/20 16:54:49 | javax.security.auth.login.LoginException: java.lang.NullPointerException
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.waitMonitorStartup(AEConnMgr.java:117)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.executeQuery(AEConnMgr.java:106)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuery.execute(AEQuery.java:94)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuerySQL.execute(AEQuerySQL.java:40)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.security.AEEiamLoginModule.getEEMServer(AEEiamLoginModule.java:561)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.security.AEEiamLoginModule.getSafeContextUsingSCF(AEEiamLoginModule.java:240)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.security.AEEiamLoginModule.getSafeContextFacade(AEEiamLoginModule.java:232)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.security.AEEiamLoginModule.authenticateWithPassword(AEEiamLoginModule.java:418)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.security.AEEiamLoginModule.login(AEEiamLoginModule.java:197)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.security.AccessController.doPrivileged(AccessController.java:770)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:418)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:332)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.authenticator.BasicAuthenticator.doAuthenticate(BasicAuthenticator.java:101)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:631)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.lang.Thread.run(Thread.java:821)
INFO   | jvm 1    | 2023/07/20 16:54:49 | 
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.security.AccessController.doPrivileged(AccessController.java:770)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:418)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:332)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.authenticator.BasicAuthenticator.doAuthenticate(BasicAuthenticator.java:101)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:631)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at java.lang.Thread.run(Thread.java:821)
INFO   | jvm 1    | 2023/07/20 16:54:49 | Exception in thread "IntCodeLoader" java.lang.NullPointerException
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.waitMonitorStartup(AEConnMgr.java:117)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEConnMgr.executeQuery(AEConnMgr.java:106)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuery.execute(AEQuery.java:94)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.dbaccess.dao.AEQuerySQL.execute(AEQuerySQL.java:40)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.common.IntCodes.loadIntCodes(IntCodes.java:136)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.common.IntCodes.<init>(IntCodes.java:46)
INFO   | jvm 1    | 2023/07/20 16:54:49 |     at com.ca.waae.aehome.AEHomeInstance$IntCodeLoader.run(AEHomeInstance.java:168)
INFO   | jvm 1    | 2023/07/20 16:55:29 | 20-Jul-2023 16:55:29.511 WARNING [https-js

 

Environment

Release : 12.0

Cause

Customer database was fully walleted.  Autosys as_server works fine too using those wallet references

 

we enabled AEWS debug: 

https://knowledge.broadcom.com/external/article?articleId=223063

and AEWS  SQL debug:  

https://knowledge.broadcom.com/external/article?articleId=226077

 

 

 

This gave us more insight this time.

023-07-21 12:51:34,026 [https-jsse-nio-9443-exec-10] DEBUG com.ca.waae.dbaccess.config.DatabaseProperties(467) - jdbc url created, url=jdbc:oracle:oci:@wallet123
2023-07-21 12:51:34,026 [https-jsse-nio-9443-exec-10] DEBUG com.ca.waae.dbaccess.dao.AEConnection(125) - Wallet location cannot be empty. Set ORACLEWALLETLOCATION to appropriate value.
2023-07-21 12:51:34,026 [https-jsse-nio-9443-exec-10] DEBUG com.ca.waae.dbaccess.dao.AEConnMgr(475) - Wallet location cannot be empty. Set ORACLEWALLETLOCATION to appropriate value.
2023-07-21 12:51:34,027 [https-jsse-nio-9443-exec-10] DEBUG com.ca.waae.aehome.AEHomeInstance(73) - Wallet location cannot be empty. Set ORACLEWALLETLOCATION to appropriate value. com.ca.waae.dbaccess.dao.AEConnMgrException: Wallet location cannot be empty. Set ORACLEWALLETLOCATION to appropriate value.
    at com.ca.waae.dbaccess.dao.AEConnMgr.createConnections(AEConnMgr.java:476)
    at com.ca.waae.dbaccess.dao.AEConnMgr.<init>(AEConnMgr.java:90)
    at com.ca.waae.aehome.AEHomeInstance.<init>(AEHomeInstance.java:67)
    at com.ca.waae.aehome.AEHome.addInstance(AEHome.java:77)
    at com.ca.waae.aehome.AEHome.<clinit>(AEHome.java:56)
    at com.ca.waae.dbaccess.dao.AEQuery.execute(AEQuery.java:92)
    at com.ca.waae.dbaccess.dao.AEQuerySQL.execute(AEQuerySQL.java:40)
    at com.ca.waae.security.AEEiamLoginModule.getEEMServer(AEEiamLoginModule.java:561)
    at com.ca.waae.security.AEEiamLoginModule.getSafeContextUsingSCF(AEEiamLoginModule.java:240)
    at com.ca.waae.security.AEEiamLoginModule.getSafeContextFacade(AEEiamLoginModule.java:232)
    at com.ca.waae.security.AEEiamLoginModule.authenticateWithPassword(AEEiamLoginModule.java:418)
    at com.ca.waae.security.AEEiamLoginModule.login(AEEiamLoginModule.java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 

Because this is a walleted installation, we need to make sure the ORACLEWALLETLOCATION environment variable needs to be available to the web server process.  We verified that it did not  via   strings /proc/<pid of AEWS>/environ

 

Resolution

To fix the issue, we edited the  $AUTOUSER/webserver/bin/waae_webserver*    shell script and exported the variable to correct value

ORACLEWALLETLOCATION=/opt/oracleclient/wallet/

export ORACLEWALLETLOCATION

 

restart web server and that fixed the issue