JBoss EAP startup vault error with APM agent and OpenJDK 1.8.0_171
Article ID: 132340
Updated On:
Products
CA Application Performance Management Agent (APM / Wily / Introscope)
INTROSCOPE
Issue/Introduction
An error is encountered when starting a JBoss EAP 7.0.9 server with APM 10.7 agent and the server will not start.
Without the required APM configuration, the server starts with no error.
ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("core-service" => "vault")]): org.jboss.as.server.services.security.VaultReaderException: WFLYSRV0076: Error initialzing vault -- org.jboss.security.vault.SecurityVaultException: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:93)
at org.jboss.as.server.services.security.VaultAddHandler.performRuntime(VaultAddHandler.java:84)
at org.jboss.as.controller.AbstractAddStepHandler.performRuntime(AbstractAddStepHandler.java:337)
at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:151)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1349)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:495)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:389)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:351)
at org.jboss.as.server.ServerService.boot(ServerService.java:402)
at org.jboss.as.server.ServerService.boot(ServerService.java:371)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:301)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.jboss.security.vault.SecurityVaultException: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:210)
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:91)
... 14 more
Caused by: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:691)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:205)
... 15 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:201)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:151)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:688)
... 16 more
FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting .See previous messages for details.
INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0050: JBoss EAP 7.0.9.GA (WildFly Core 2.1.20.Final-redhat-1) stopped
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:93)
at org.jboss.as.server.services.security.VaultAddHandler.performRuntime(VaultAddHandler.java:84)
at org.jboss.as.controller.AbstractAddStepHandler.performRuntime(AbstractAddStepHandler.java:337)
at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:151)
at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:890)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1349)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:495)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:389)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:351)
at org.jboss.as.server.ServerService.boot(ServerService.java:402)
at org.jboss.as.server.ServerService.boot(ServerService.java:371)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:301)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.jboss.security.vault.SecurityVaultException: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:210)
at org.jboss.as.server.services.security.RuntimeVaultReader.createVault(RuntimeVaultReader.java:91)
... 14 more
Caused by: java.lang.RuntimeException: PBOX00140: Unable to get keystore (/opt/jboss/eap-7/newtron/hc/vault/vault.keystore)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:691)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:205)
... 15 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:201)
at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:151)
at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:688)
... 16 more
FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting .See previous messages for details.
INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0050: JBoss EAP 7.0.9.GA (WildFly Core 2.1.20.Final-redhat-1) stopped
Without the required APM configuration, the server starts with no error.
Environment
Jboss EAP 7.0.9
APM 10.7
OpenJDk 1.8.0_171
APM 10.7
OpenJDk 1.8.0_171
Cause
There is a published issue with JBoss EAP 7 and OpenJDK related to how the JCEKS keystore loads its keys in OpenJDK
https://access.redhat.com/solutions/3419621
The workaround is to configure the system to allow access to com.sun.crypto.provider
In this scenario, the required workaround was already in the Jboss configuration domain.conf file:
However, as part of the APM configuration, we also require to add reference to our packages and provide the suggestion of the JVM argument. This takes into consideration org.jboss.byteman, but not the extra property:
This JVM configuration overrides the setting in domain.conf, so the com.sun.crypto.provider access is not provided, hence causing the problem.
https://access.redhat.com/solutions/3419621
The workaround is to configure the system to allow access to com.sun.crypto.provider
In this scenario, the required workaround was already in the Jboss configuration domain.conf file:
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.sun.crypto.provider"
However, as part of the APM configuration, we also require to add reference to our packages and provide the suggestion of the JVM argument. This takes into consideration org.jboss.byteman, but not the extra property:
-Djboss.modules.system.pkgs=org.jboss.byteman,com.wily,com.wily.*
This JVM configuration overrides the setting in domain.conf, so the com.sun.crypto.provider access is not provided, hence causing the problem.
Resolution
The reference to allow access to com.sun.crypto.provider was combined with the required reference to APM packages in the JVM argument
-Djboss.modules.system.pkgs=org.jboss.byteman,com.wily,com.wily.*,com.sun.crypto.provider
Feedback
Yes
No
Powered by
