JBoss EAP startup vault error with APM agent and OpenJDK 1.8.0_171

book

Article ID: 132340

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

An error is encountered when starting a JBoss EAP 7.0.9 server with APM 10.7 agent and the server will not start.
 
ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("core-service" => "vault")]): org.jboss.as.server.services.security.VaultReaderException: WFLYSRV0076: Error initialzing vault -- org.jboss.security.vault.SecurityVaultException: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:93)
at org.jboss.as.server.services.security.VaultAddHandler.performRuntime(VaultAddHandler.java:84)
at org.jboss.as.controller.AbstractAddStepHandler.performRuntime(AbstractAddStepHandler.java:337)
at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:151)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1349)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:495)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:389)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:351)
at org.jboss.as.server.ServerService.boot(ServerService.java:402)
at org.jboss.as.server.ServerService.boot(ServerService.java:371)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:301)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.jboss.security.vault.SecurityVaultException: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore) 
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:210)
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:91)
 ... 14 more
Caused by: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:691)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:205)
... 15 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:201)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:151)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:688)
... 16 more
FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting .See previous messages for details.
INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0050: JBoss EAP 7.0.9.GA (WildFly Core 2.1.20.Final-redhat-1) stopped

Without the required APM configuration, the server starts with no error.

Cause

There is a published issue with JBoss EAP 7 and OpenJDK related to how the JCEKS keystore loads its keys in OpenJDK

 https://access.redhat.com/solutions/3419621

The workaround is to configure the system to allow access to com.sun.crypto.provider

In this scenario, the required workaround was already in the Jboss configuration domain.conf file:
 
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.sun.crypto.provider"

However, as part of the APM configuration, we also require to add reference to our packages and provide the suggestion of the JVM argument. This takes into consideration org.jboss.byteman, but not the extra property:
 
-Djboss.modules.system.pkgs=org.jboss.byteman,com.wily,com.wily.*

This JVM configuration overrides the setting in domain.conf, so the com.sun.crypto.provider access is not provided, hence causing the problem.

Environment

Jboss EAP 7.0.9
APM 10.7
OpenJDk 1.8.0_171

Resolution

The reference to allow access to com.sun.crypto.provider was combined with the required reference to APM packages in the JVM argument
 
-Djboss.modules.system.pkgs=org.jboss.byteman,com.wily,com.wily.*,com.sun.crypto.provider