Proxy error while accessing applications tab from portal ui

Products

CA API Developer Portal CA API Gateway

Issue/Introduction

Proxy Error after clicking on applications tab on portal

Environment

Release: L7APIP99000-3.1-API Developer Portal-Perpetual
Component:

Cause

When you click on Applications tab portal would throw below error.

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /dashboard/organizations/applications.

Reason: Error reading from remote server

The blow stack trace indicates that the system was not able to determine the organization admin for every organization referenced by the lrsaccountplanrequest table. There are two possibilities, either the organization no longer exists or the organization does not have a user with the organizationAdmin role.

Below is the stack trace from catalina.out

ERROR (http-nio-37080-exec-132:) - [AccountPlanRequestXS general] -- com.thelevel.cms.services.exceptions.ServiceException	at com.thelevel.lrs.daoservices.entity.OrganizationEntityServices.getOrganizationAdmin(OrganizationEntityServices.java:382)	at com.thelevel.lrs.services.organization.OrganizationSourceServices.getOrganizationAdmin(OrganizationSourceServices.java:200)	at com.thelevel.lrs.bm.AccountPlanRequestXS.listRequests(AccountPlanRequestXS.java:300)	at com.thelevel.lrs.bm.AccountPlanRequestXS.getLRSXML(AccountPlanRequestXS.java:133)	at com.thelevel.lrs.AbstractLRSSource.getXML(AbstractLRSSource.java:97)	at com.thelevel.cms.xmlsources.AbstractXMLSource.getXML(AbstractXMLSource.java:172)	at com.thelevel.cms.site.PageRule.handleRequest(PageRule.java:277)	at com.thelevel.cms.site.CMSSiteHandler.handleRequest(CMSSiteHandler.java:488)	at com.thelevel.cms.site.CMSSiteHandler.handleRequest(CMSSiteHandler.java:439)	at com.thelevel.cms.site.SiteServlet.processRequest(SiteServlet.java:423)	at com.thelevel.cms.site.SiteServlet.instrumentRequest(SiteServlet.java:223)	at com.thelevel.cms.site.SiteServlet.filterRequest(SiteServlet.java:177)	at com.thelevel.cms.site.SiteServlet.service(SiteServlet.java:127)	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at com.thelevel.cms.admin.JSONFilter.doFilter(JSONFilter.java:40)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at com.thelevel.lrs.JForumFilter.doFilter(JForumFilter.java:108)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at com.thelevel.cms.identity.AuthFilter.doFilter(AuthFilter.java:106)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.picocontainer.web.PicoServletContainerFilter.doFilter(PicoServletContainerFilter.java:74)	at com.thelevel.cms.CMSBootstrapper$ServletFilter.doFilter(CMSBootstrapper.java:649)	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)	at java.lang.Thread.run(Thread.java:748)

Resolution

Below will determine the requests where the organization does not exist:

select req.* from lrsaccountplanrequest req left join cmsOrganization org on org.orgid = req.orgid where org.name is null;

Below will determine the requests where the organization does not have an user with organizationAdmin role

select req.*, (select u.username from cmsOrganization org join cmsorganizationusers orguser on org.orgid = orguser.orgid join cmsUserRoles ur on ur.uid = orguser.userid join cmsRoles r on r.id = ur.roleId join cmsUsers u on orguser.userid = u.id where org.orgid = req.orgid and r.name = 'organizationAdmin') orgAdmin from lrsaccountplanrequest req having orgAdmin is null;

Below is the sample output

The query results prove that 2 organizations have been deleted. The only solution is to delete the account plan change request or set them to Rejected.

update lrsaccountplanrequest set status = 'Rejected' where id in (3,5,7,9) and status = 'Requested';

* NOTE * :Before making any changes to portal DB kindly have DB backup and snapshot of the machine in place as a safety measure