Trying to restart wildfly server however it is hung and never starts up. The following message is shown repeatedly in DEBUG level logs:
2018-06-06 15:32:52,599 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name
2018-06-06 15:32:52,599 DEBUG [org.hornetq.jms.server] (Periodic Recovery) looking for recover at DelegatingSession [session=ClientSessionImpl [name=####-####-####_####, username=null, closed=false, factory = ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=bd3acfef-69c2-11e8-ab6a-e13f1e5ad8d1, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=#####-####-####-####, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null], metaData=()]@#######] configuration [XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=####-####-####_####, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=****]]
2018-06-06 15:33:02,600 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name id2lx1017
2018-06-06 15:33:02,600 DEBUG [org.hornetq.jms.server] (Periodic Recovery) looking for recover at DelegatingSession [session=ClientSessionImpl [name=####-####-####_####, username=null, closed=false, factory = ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=bd3acfef-69c2-11e8-ab6a-e13f1e5ad8d1, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=bd3acfef-69c2-11e8-ab6a-e13f1e5ad8d1, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null], metaData=()]@#######] configuration [XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=bd3acfef-69c2-11e8-ab6a-e13f1e5ad8d1, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=****]]
2018-06-06 15:33:05,264 DEBUG [org.hornetq.core.server] (hornetq-expiry-reaper-thread) Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue
Other times, we just see the server starting without ever attempting to deploy
2021-09-28 03:45:39,902 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final
2021-09-28 03:45:40,553 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.2.Final
2021-09-28 03:45:40,608 INFO [org.jboss.as] (MSC service thread 1-5) JBAS015899: WildFly 8.2.1.Final "Tweek" starting
2021-09-28 03:45:40,614 DEBUG [org.jboss.as.config] (MSC service thread 1-5) Configured system properties:
awt.toolkit = sun.awt.windows.WToolkit
com.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager = true
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.headless = true
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = D:\wildfly-8.2.1.Final\jboss-modules.jar
java.class.version = 52.0
java.endorsed.dirs = C:\Program Files\Java\jdk1.8.0_291\jre\lib\endorsed
java.ext.dirs = C:\Program Files\Java\jdk1.8.0_291\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home = C:\Program Files\Java\jdk1.8.0_291\jre
java.io.tmpdir = C:\Windows\TEMP\
java.library.path = C:\Program Files\Java\jdk1.8.0_291\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files\dotnet\;C:\Program Files (x86)\dotnet\;C:\Program Files\PowerShell\6\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\Java\jdk;C:\Program Files\Java\jre;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;.
java.net.preferIPv4Stack = true
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_291-b10
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.8
java.util.logging.manager = org.jboss.logmanager.LogManager
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.8.0_291
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.8
java.vm.vendor = Oracle Corporation
java.vm.version = 25.291-b10
javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
javax.net.ssl.trustStore = D:\wildfly-8.2.1.Final\bin\cacert
javax.net.ssl.trustStorePassword = {password}
javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
jboss.as.management.blocking.timeout = 900
jboss.bind.address = 0.0.0.0
jboss.home.dir = D:\wildfly-8.2.1.Final
jboss.host.name = dev-idm-2016
jboss.modules.dir = D:\wildfly-8.2.1.Final\modules
jboss.modules.system.pkgs = org.jboss.byteman
jboss.node.name = dev-idm-2016
jboss.qualified.host.name = dev-idm-2016
jboss.server.base.dir = D:\wildfly-8.2.1.Final\standalone
jboss.server.config.dir = D:\wildfly-8.2.1.Final\standalone\configuration
jboss.server.data.dir = D:\wildfly-8.2.1.Final\standalone\data
jboss.server.deploy.dir = D:\wildfly-8.2.1.Final\standalone\data\content
jboss.server.log.dir = D:\wildfly-8.2.1.Final\standalone\log
jboss.server.name = dev-idm-2016
jboss.server.persist.config = true
jboss.server.temp.dir = D:\wildfly-8.2.1.Final\standalone\tmp
line.separator =
logging.configuration = file:D:\wildfly-8.2.1.Final\standalone\configuration/logging.properties
module.path = D:\wildfly-8.2.1.Final\modules
nete.j2ee.vendor = jboss
org.jboss.boot.log.file = D:\wildfly-8.2.1.Final\standalone\log\server.log
org.jboss.resolver.warning = true
org.xml.sax.driver = __redirected.__XMLReaderFactory
os.arch = amd64
os.name = Windows Server 2016
os.version = 10.0
path.separator = ;
program.name = standalone.bat
sun.arch.data.model = 64
sun.boot.class.path = C:\Program Files\Java\jdk1.8.0_291\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\rt.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\sunrsasign.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_291\jre\classes
sun.boot.library.path = C:\Program Files\Java\jdk1.8.0_291\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.java.command = D:\wildfly-8.2.1.Final\jboss-modules.jar -mp D:\wildfly-8.2.1.Final\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\wildfly-8.2.1.Final -c standalone-full.xml -b 0.0.0.0 -Dnete.j2ee.vendor=jboss -Djboss.as.management.blocking.timeout=900
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = Cp1252
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level =
user.country = US
user.dir = D:\wildfly-8.2.1.Final\bin
user.home = C:\Windows\system32\config\systemprofile
user.language = en
user.name = DEV-IDM-2016$
user.script =
user.timezone = UTC
user.variant =
2021-09-28 03:45:40,620 DEBUG [org.jboss.as.config] (MSC service thread 1-5) VM Arguments: -Dprogram.name=standalone.bat -Xms8196m -Xmx8196m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.awt.headless=true -Dcom.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager=true -Djava.net.preferIPv4Stack=true -Djavax.net.ssl.trustStore=D:\wildfly-8.2.1.Final\bin\cacert -Djavax.net.ssl.trustStorePassword={password} -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.boot.log.file=D:\wildfly-8.2.1.Final\standalone\log\server.log -Dlogging.configuration=file:D:\wildfly-8.2.1.Final\standalone\configuration/logging.properties
2021-09-28 03:45:44,166 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015950: WildFly 8.2.1.Final "Tweek" stopped in 6ms
2021-09-28 03:45:44,696 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final
2021-09-28 03:45:44,838 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.2.Final
2021-09-28 03:45:44,886 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: WildFly 8.2.1.Final "Tweek" starting
2021-09-28 03:45:44,886 DEBUG [org.jboss.as.config] (MSC service thread 1-7) Configured system properties:
awt.toolkit = sun.awt.windows.WToolkit
com.sun.jersey.server.impl.cdi.lookupExtensionInBeanManager = true
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.headless = true
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = D:\wildfly-8.2.1.Final\jboss-modules.jar
java.class.version = 52.0
java.endorsed.dirs = C:\Program Files\Java\jdk1.8.0_291\jre\lib\endorsed
java.ext.dirs = C:\Program Files\Java\jdk1.8.0_291\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home = C:\Program Files\Java\jdk1.8.0_291\jre
java.io.tmpdir = C:\Windows\TEMP\
java.library.path = C:\Program Files\Java\jdk1.8.0_291\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files\dotnet\;C:\Program Files (x86)\dotnet\;C:\Program Files\PowerShell\6\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\Java\jdk;C:\Program Files\Java\jre;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;.
java.net.preferIPv4Stack = true
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_291-b10
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.8
java.util.logging.manager = org.jboss.logmanager.LogManager
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.8.0_291
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.8
java.vm.vendor = Oracle Corporation
java.vm.version = 25.291-b10
javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
javax.net.ssl.trustStore = D:\wildfly-8.2.1.Final\bin\cacert
javax.net.ssl.trustStorePassword = {password}
javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
jboss.as.management.blocking.timeout = 900
jboss.bind.address = 0.0.0.0
jboss.home.dir = D:\wildfly-8.2.1.Final
jboss.host.name = dev-idm-2016
jboss.modules.dir = D:\wildfly-8.2.1.Final\modules
jboss.modules.system.pkgs = org.jboss.byteman
jboss.node.name = dev-idm-2016
jboss.qualified.host.name = dev-idm-2016
jboss.server.base.dir = D:\wildfly-8.2.1.Final\standalone
jboss.server.config.dir = D:\wildfly-8.2.1.Final\standalone\configuration
jboss.server.data.dir = D:\wildfly-8.2.1.Final\standalone\data
jboss.server.deploy.dir = D:\wildfly-8.2.1.Final\standalone\data\content
jboss.server.log.dir = D:\wildfly-8.2.1.Final\standalone\log
jboss.server.name = dev-idm-2016
jboss.server.persist.config = true
jboss.server.temp.dir = D:\wildfly-8.2.1.Final\standalone\tmp
line.separator =
logging.configuration = file:D:\wildfly-8.2.1.Final\standalone\configuration/logging.properties
module.path = D:\wildfly-8.2.1.Final\modules
nete.j2ee.vendor = jboss
org.jboss.boot.log.file = D:\wildfly-8.2.1.Final\standalone\log\server.log
org.jboss.resolver.warning = true
org.xml.sax.driver = __redirected.__XMLReaderFactory
os.arch = amd64
os.name = Windows Server 2016
os.version = 10.0
path.separator = ;
program.name = standalone.bat
sun.arch.data.model = 64
sun.boot.class.path = C:\Program Files\Java\jdk1.8.0_291\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\rt.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\sunrsasign.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_291\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_291\jre\classes
sun.boot.library.path = C:\Program Files\Java\jdk1.8.0_291\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.java.command = D:\wildfly-8.2.1.Final\jboss-modules.jar -mp D:\wildfly-8.2.1.Final\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\wildfly-8.2.1.Final -c standalone-full.xml -b 0.0.0.0 -Dnete.j2ee.vendor=jboss -Djboss.as.management.blocking.timeout=900
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = Cp1252
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level =
user.country = US
user.dir = D:\wildfly-8.2.1.Final\bin
user.home = C:\Windows\system32\config\systemprofile
user.language = en
user.name = DEV-IDM-2016$
user.script =
user.timezone = UTC
user.variant =
2021-09-28 04:01:05,038 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.2.1.Final "Tweek" started in 7015ms - Started 276 of 327 services (106 services are lazy, passive or on-demand)
Release: 14.x
Component: IDMGR
Wildfly server looks for this file with extension ".dodeploy" to deploy any ear files. We needed to create the file which is missing.
Sometimes if Wildfly fails to deploy ear at server start, it will create file with extension “.undeployed”. We need to update extension to force deployment.
The deployment folder should contain the following 2 files, for the server to trigger deployment
/JBOSS_HOME/standalone/deployments/iam_im.ear.dodeploy
/JBOSS_HOME/standalone/deployments/castylesr5.1.1.ear.dodeploy