uimapi and webservices_rest stop working after upgrading to 20.3.3 or 20.4: Pool not open

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

After upgrading to UIM 20.3.3 REST API calls  are failing with HTTP 500 error. This is the exception as reported in the browser when attempting an API call:

java.lang.IllegalStateException: Pool not open	
org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:736)

The failure happens with both uimapi and webservices_rest deployed to the OC robot, and the uimapi log has the exact same info the browser shows. 

2021-04-13 13:07:46,922 ERROR com.ca.uimapi.services.controller.AccountResource:getAccountsV0:123 [http-nio-80-exec-20]   - getAccounts failed: 
java.lang.IllegalStateException: Pool not open
 at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:736)
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:414)
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:365)
 at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
 at com.nimsoft.db.PooledDataSourceWrapper.getConnection(PooledDataSourceWrapper.java:33)

If using webservices_rest with a GET call direct in a URL without a REST client it might just return a http 404 error. 

When using a REST client it will show up in the return data. 

In some Scenarios, only specific QOS metrics calls, or account calls are failing when using webservices_rest while uimapi keeps working fine:

Example Error: 

ERROR [http-nio-80-exec-9, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/rest].[webservices_rest]] invoke() Servlet.service() for servlet [webservices_rest] in context with path [/rest] threw exception
toukokuuta 19 12:40:00:451 ERROR [http-nio-80-exec-9, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/rest].[webservices_rest]] java.lang.IllegalStateException: Pool not open
 at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:736)
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:414)
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:365)
 at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
 at com.nimsoft.db.PooledDataSourceWrapper.getConnection(PooledDataSourceWrapper.java:33)
 at com.nimsoft.nms.nimwscommon.dao.Util.getConnection(Util.java:45)
 at com.nimsoft.nms.nimwscommon.controller.QoSController.getConnection(QoSController.java:47)
 at com.nimsoft.nms.restfulws.services.QosResource.getDefinition(QosResource.java:62)
 at sun.reflect.GeneratedMethodAccessor1137.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
 at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

Environment

Release : UIM 20.3.x, UIM 20.4

Component : UIM - WEBSERVICES_REST

Cause

The failure is caused by old packages at the OC robot. 

Compare the installed packages to this list:
uim_reportscheduler 20.33hf2
ump 20.33
ump_accountadmin 20.33hf2
ump_cabi 4.24hf2
ump_dashboard 20.33hf1
ump_operatorconsole 2.10hf3
ump_slm 20.33hf2 
wasp 20.33

Any that are version 9.x, or 8.x are to be suspected as deprecated and should be removed. 

webapps known to cause the problem: dapi, jasperserver_pro, umpjslib, webalarms, HtmlAlarmConsole, umpmeadia, maintenancemode (OC/ump webapp)

Note: jasperserver_pro will only show up in wasp raw configure or cfg under the webapps section. 

Resolution

Remove the old packages from the OC robot. 

UIM How to remove deprecated webapps from wasp
Article Id: 212899
https://knowledge.broadcom.com/external/article?articleId=212899

Additional Information

uimapi log location: OC robot > $)\Nimsoft\probes\service\wasp\uimapi.log 

UIM / DX IM - How to see all the packages deployed to a robot
Article Id: 141322
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=141322

The obsolete webapps are documented here:
20.3 Operator Console Installer - Webapps Considerations