HotswapProjects/user

Graziano Felline

@ogipogi

After applying the fix i get again the originally `NoClassDefFoundError``

Formatter ##CONSOLE-FORMATTER## is not defined
HOTSWAP AGENT: 12:56:49.703 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.2-SNAPSHOT} - unlimited runtime class redefinition.
HOTSWAP AGENT: 12:56:50.527 INFO (org.hotswap.agent.config.PluginRegistry) - Discovered plugins: [Hotswapper, WatchResources, AnonymousClassPatch, ClassInitPlugin, JdkPlugin, Proxy, WebObjects, Weld, JBossModules, Undertow]
Listening for transport dt_socket at address: 8787
Exception in thread "main" java.lang.NoClassDefFoundError: org/hotswap/agent/util/classloader/HotswapAgentClassLoaderExt
    at org.jboss.modules.Module.<init>(Module.java:247)
    at org.jboss.modules.ModuleLoader$3.run(ModuleLoader.java:729)
    at org.jboss.modules.ModuleLoader$3.run(ModuleLoader.java:724)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.jboss.modules.ModuleLoader.defineModule(ModuleLoader.java:724)
    at org.jboss.modules.ModuleLoader.loadModuleLocal(ModuleLoader.java:531)
    at org.jboss.modules.DelegatingModuleLoader.preloadModule(DelegatingModuleLoader.java:57)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:295)
    at __redirected.__RedirectedUtils.loadProvider(__RedirectedUtils.java:92)
    at __redirected.__RedirectedUtils.loadProvider(__RedirectedUtils.java:86)
    at __redirected.__DocumentBuilderFactory.changeDefaultFactory(__DocumentBuilderFactory.java:53)
    at __redirected.__JAXPRedirected.changeAll(__JAXPRedirected.java:53)
    at org.jboss.modules.Main.main(Main.java:428)

Graziano Felline

@ogipogi

I managed to avoid the NoClassDefFoundError with adding
-Xbootclasspath/p:$JBOSS_HOME/hotswap-agent/hotswap-agent-1.4.2-SNAPSHOT.jar \to JAVA_OPTS

I get now new Exception during deploy of activemq-rar and Camunda Engine that i need to investigate first.

13:29:58,467 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0002: Bound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
13:30:00,040 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.module.service."deployment.activemq-rar.rar".main: org.jboss.msc.service.StartException in service jboss.module.service."deployment.activemq-rar.rar".main: Failed to start service
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
    at org.hotswap.agent.config.PluginRegistry.getPluginClass(PluginRegistry.java:293)
    at org.hotswap.agent.config.PluginRegistry.initializePlugin(PluginRegistry.java:165)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:300)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:283)
    at org.jboss.as.server.moduleservice.ModuleLoadService.start(ModuleLoadService.java:66)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
    ... 6 more

13:30:00,134 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.module.service."deployment.camunda-webapp-ee-jboss-7.11.11-ee.war".main: org.jboss.msc.service.StartException in service jboss.module.service."deployment.camunda-webapp-ee-jboss-7.11.11-ee.war".main: Failed to start service
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
    at org.hotswap.agent.config.PluginRegistry.getPluginClass(PluginRegistry.java:293)
    at org.hotswap.agent.config.PluginRegistry.initializePlugin(PluginRegistry.java:165)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:300)
    at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:283)
    at org.jboss.as.server.moduleservice.ModuleLoadService.start(ModuleLoadService.java:66)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
    ... 6 more

The JBossModules plugin is now loaded:

HOTSWAP AGENT: 13:29:53.398 INFO (org.hotswap.agent.config.PluginRegistry) - Discovered plugins: [Hotswapper, WatchResources, AnonymousClassPatch, ClassInitPlugin, JdkPlugin, Owb, Proxy, WebObjects, Weld, JBossModules, Undertow]

I need now to understand this NullPointerException

Caused by: java.lang.NullPointerException
    at org.hotswap.agent.config.PluginRegistry.getPluginClass(PluginRegistry.java:293)

Vladimir Dvorak

@skybber

Probably system class loader is null

Vladimir Dvorak

@skybber

Probably correct version of PluginRegistry.getPluginClass for null bootstrap classloader is:

protected Class<Object> getPluginClass(String pluginClass) {
    try {
        // noinspection unchecked
        if (getClass().getClassLoader() == null) {
            return (Class<Object>) Class.forName(pluginClass, true, null);
        }
        return (Class<Object>) getClass().getClassLoader().loadClass(pluginClass);
    } catch (ClassNotFoundException e) {
        throw new IllegalArgumentException("Plugin class not found " + pluginClass, e);
    }
}

I've never investigated why the bootstrap class loader could be null, since I had never faced to this problem

Graziano Felline

@ogipogi

Thank You Vladimir for the additional infos

Vladimir Dvorak

@skybber

If it fix your problem, we can add the fix to ha-core

Graziano Felline

@ogipogi

I will test this

Vladimir Dvorak

@skybber

thanks

I don't have JbossEAP at all ...

Graziano Felline

@ogipogi

:)
I have problems to build a local SNAPSHOT jar

Vladimir Dvorak

@skybber

OK, I'll prepare it

Graziano Felline

@ogipogi

Thank you very much

Vladimir Dvorak

@skybber

snapshot updated

Graziano Felline

@ogipogi

ok, I'll test it

Vladimir Dvorak

@skybber

btw, you need dcevm11 to build HA

Graziano Felline

@ogipogi

Ahh, that was maybe the issue

Vladimir Dvorak

@skybber

I'm using dcevm11 as system jdk now for a week

https://github.com/TravaOpenJDK/trava-jdk-11-dcevm-staged

Graziano Felline

@ogipogi

I tested it:

activemq is deployed
New exception during deployment of Camunda engine

14:46:08,971 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 69) MSC000001: Failed to start service jboss.deployment.unit."camunda-webapp-ee-jboss-7.11.11-ee.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."camunda-webapp-ee-jboss-7.11.11-ee.war".undertow-deployment: java.lang.NoClassDefFoundError: io/undertow/server/handlers/resource/ResourceManager
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.NoClassDefFoundError: io/undertow/server/handlers/resource/ResourceManager
    at java.lang.ClassLoader.findBootstrapClass(Native Method)
    at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1009)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:408)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.findSystemClass(ClassLoader.java:998)
    at org.jboss.modules.ConcurrentClassLoader.findSystemClassInternal(ConcurrentClassLoader.java:416)
    at org.jboss.modules.JDKSpecific.getSystemClass(JDKSpecific.java:180)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:395)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
    ... 8 more

I need to investigate what happened

Vladimir Dvorak

@skybber

You can try to disable UndertowPlugin

Graziano Felline

@ogipogi

I'm testing now exactly that :)

Vladimir Dvorak

@skybber

UndertowPlugin is for resource loading

Graziano Felline

@ogipogi

It looks good, the exception disappear.
Thank you very much Vladimir.
I'll test now the hot swapping.

Vladimir Dvorak

@skybber

OK, may be Undertow will be needed

then something is wrong in UndertowTransformer - PrefixingResourceManager is extended from jboss ResourceManager, that is not found at

at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)

Graziano Felline

@ogipogi

Ok, i get additional Exception during deployment of EAR file, also i need to investigate.

15:02:26,199 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.deployment.unit."app.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."app.ear".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "app.ear"
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:151)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Error: java.lang.NullPointerException
    at org.jboss.as.weld.deployment.BeanDeploymentArchiveImpl.<init>(BeanDeploymentArchiveImpl.java:111)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.createExplicitBeanDeploymentArchive(BeanArchiveProcessor.java:347)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.processResourceRoot(BeanArchiveProcessor.java:304)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.handleResourceRoot(BeanArchiveProcessor.java:237)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.access$100(BeanArchiveProcessor.java:206)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor.deploy(BeanArchiveProcessor.java:113)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:144)
    ... 8 more
Caused by: java.lang.NullPointerException
    ... 15 more

15:02:26,213 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "app.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"app.ear\".POST_MODULE" => "WFLYSRV0153: Failed to process phase POST_MODULE of deployment \"app.ear\"
    Caused by: java.lang.Error: java.lang.NullPointerException
    Caused by: java.lang.NullPointerException"}}

During deployment of submodules i get now several new HA Info message from JBossModules plugin, this looks good.

HOTSWAP AGENT: 15:02:25.754 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.jbossmodules.JBossModulesPlugin' initialized in ClassLoader 'ModuleClassLoader for Module "deployment.app.ear.my-module-a-impl.jar" from Service Module Loader'.

Graziano Felline

@ogipogi

Ok, this errors during deploy of EAR file are related with the last fix ( bootstrap class loader). If i deactivate Undertow and JBossModules i get the same Exception...

Vladimir Dvorak

@skybber

INFO messages are ok

bootstrap fix seem ok for me

have you rest of stack trace - pby there is part related to HA

you can try to disable WeldPlugin too

Graziano Felline

@ogipogi

ok, the exception disappear if i disable also the weld plugin

Vladimir Dvorak

@skybber

We have tests for wilfly up to 3.1.3
https://github.com/HotswapProjects/HotswapAgent/blob/master/plugin/hotswap-agent-weld-plugin/run-tests.sh#L79

But not in EAP

Graziano Felline

@ogipogi

ok, thank you for the info

Vladimir Dvorak

@skybber

not at all, hope it helps you

Graziano Felline

@ogipogi

:thumbsup:

Graziano Felline

@ogipogi

Ok, it looks lie the app i have is to large (large monolith).
The deployment of ear hangs, if i have JBossModules plugin activated.
It hangs on:

17:40:09,444 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment app.ear

jstack:

at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResource(ExternalBeanArchiveProcessor.java:330)
    at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResources(ExternalBeanArchiveProcessor.java:313)

With disabled JBossModules plugin deployment duration is ca. 8 min.

Any hints how i can patch JBossModules plugin for test if swapping works to an explicit module (ejb jar)?

Vladimir Dvorak

@skybber

look into ModuleClassLoaderTransformer

if you are not attached by external debugger, then you also need autoHotswap=true

There are generally 2 ways how the JVM can get new bytecode, 1. from debugger 2. from HA ussing autoHotswap

Vladimir Dvorak

@skybber

Anyway for Wildfly, I would try to remove unused plugins directly from hotswap-agent.jar. It will avoid a lot of useless scanning especially on jboss eap, where are a lot of classloaders that slow things down.

1 reply

I would recommend to test it outside docker at first, if it works, then docker is next level

Graziano Felline

@ogipogi

The reason for hanging (much slower deployment) is:

Setup has a lot of sub projects (large monolith)
I set hotswap-agent.properties in one of these sub projects src/main/resources folder withextraClasspath=[path_to_my_project]/build/classes/java/main
During deployment of other ejbs (other sub project) it checks now if on the extraClasspath it exists a META-INF/beans.xml or META-INF/jandex.idx file

Stack (java.nio.file.Files.exists(Files.java:2385))

java.lang.Thread.State: RUNNABLE
  at sun.nio.fs.UnixNativeDispatcher.access0(Native Method)
  at sun.nio.fs.UnixNativeDispatcher.access(UnixNativeDispatcher.java:449)
  at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:306)
  at java.nio.file.Files.exists(Files.java:2385)
  at org.jboss.modules.PathResourceLoader.lambda$getResource$4(PathResourceLoader.java:155)
  at org.jboss.modules.PathResourceLoader$$Lambda$7/2033524545.run(Unknown Source)
  at org.jboss.modules.PathResourceLoader.doPrivilegedIfNeeded(PathResourceLoader.java:248)
  at org.jboss.modules.PathResourceLoader.getResource(PathResourceLoader.java:155)
  at org.jboss.modules.ModuleClassLoader.loadResourceLocal(ModuleClassLoader.java:410)
  at org.jboss.modules.ModuleClassLoader$1.loadResourceLocal(ModuleClassLoader.java:144)
  at org.jboss.modules.Module.getResources(Module.java:861)
  at org.jboss.modules.Module.getExportedResources(Module.java:904)
  at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResource(ExternalBeanArchiveProcessor.java:330)
  at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResources(ExternalBeanArchiveProcessor.java:313)

Is this expected?
Without hotswap-agent.properties file it doesn't hang anymore, but i have no chance to link to classes for swapping.

Vladimir Dvorak

@skybber

I didn't see that

Graziano Felline

@ogipogi

Hi Vladimir, which part don't you see?

Vladimir Dvorak

@skybber

I did not see the problem yet

Where communities thrive

People

Repo info

Activity