Update to 6.9.2 - HTTP Status 400 - Bad request (Message Request header is too large)

Products

• CA Release Automation - Release Operations Center (Nolio)
• CA Release Automation - DataManagement Server (Nolio)

Issue/Introduction

After the upgrade of Nolio from 6.9.1 to 6.9.2 the operation completed successfully but during 
the test validation phase all the deployments were in block state in the init phase.

Besides this kind of error for the deployments was found:

Error
HTTP Status 400 - Bad Request

Type 
Exception Report

Message 
Request header is too large.

Description
The server cannot or will not process the request due to something tha is perceived to be a client
error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).

Exception
java.lang.IllegalArgumentException: Request header is too large
 at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:593)
 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:293)
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1792)
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
 at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189)
 at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
 at java.lang.Thread.run(Thread.java:748)

Note
The full stack trace os the roor cause is available in the server's logs.

Apache Tomcat/9.0.105

 

Same exception was reported in catalina.out.

Note: Further occurrences of HTTP request parsing errors will be logged at DEBUG level.

Environment

CA Release Automation - version 6.9

Resolution

After had rolled back to 6.9.1, proceed with the update to 6.9.2 following the steps below:

1- Stop the NAC

2- Empty directories below under <RA_HOME>/

files
files_cache
files_registry
files_temp
persistency
activemq-data/nac/LevelDB (NAC)

3- Do the upgrade

4- Edit file <RA_HOME>/conf/server.xml and add parameter maxHttpHeaderSize in Connector port 8080 and 8443 :

    <Connector port="8080"
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxHttpHeaderSize="16384"

   <Connector port="8443"
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxHttpHeaderSize="16384"
               compression="on"
               compressionMinSize="102400"

5- Also do this before the upgrade to have more information in case of problem:

Edit file <RA_HOME>/webapps/datamanagement/WEB-INF/log4j2.properties and replace log level 
from INFO to DEBUG for these 5 lines (in bold) :

appender.RegularDMAppender.policies.size.size=20Mb
appender.RegularDMAppender.strategy.type=DefaultRolloverStrategy
appender.RegularDMAppender.strategy.max=10
appender.RegularDMAppender.strategy.fileIndex=min
...
...
# nolio all
logger.NolioAll.name=com.nolio
logger.NolioAll.level=DEBUG
logger.NolioAll.appenderRef.RegularDMAppender.ref=RegularDMAppender
logger.NolioAll.additivity=false
logger.NolioHibernate.name=com.nolio.hibernate
logger.NolioHibernate.level=INFO
logger.NolioHibernate.additivity=false
logger.NolioHibernate.appenderRef.RegularDMAppender.ref=RegularDMAppender
logger.JDBCSQLTiming.name=jdbc.sqltiming
logger.JDBCSQLTiming.level=FATAL
logger.JDBCSQLTiming.appenderRef.RegularDMAppender.ref=RegularDMAppender
logger.JDBCSQLTiming.additivity=false
logger.MonitoringThread.name=MonitoringThread
logger.MonitoringThread.level=DEBUG
logger.MonitoringThread.appenderRef.MonitoringThreadAppender.ref=MonitoringThreadAppender
logger.MonitoringThread.additivity=false
logger.ServersConnectivityScannerImpl.name=com.nolio.platform.server.dataservices.services.execmng.ServersConnectivityScannerImpl
logger.ServersConnectivityScannerImpl.level=DEBUG

logger.ExecutionStatusServiceImpl.name=com.nolio.platform.server.dataservices.services.execution.ExecutionStatusServiceImpl
logger.ExecutionStatusServiceImpl.level=DEBUG
logger.ExecutionStatusServiceImpl.appenderRef.StatusesServiceAppender.ref=StatusesServiceAppender
logger.ExecutionStatusServiceImpl.additivity=false
 

6- Restart NAC service 

cd <RA_HOME>
./nolio_server.sh stop
./nolio_server.sh start

7- Check if problem is resolved.

8- If problem persists, contact Broadcom Support Portal  Broadcom Support Portal, open a case and attach a zip of directory <RA_HOME>/logs.