HotswapProjects/user

Vladimir Dvorak

@skybber

Probably system class loader is null

Vladimir Dvorak

@skybber

Probably correct version of PluginRegistry.getPluginClass for null bootstrap classloader is:

protected Class<Object> getPluginClass(String pluginClass) {
    try {
        // noinspection unchecked
        if (getClass().getClassLoader() == null) {
            return (Class<Object>) Class.forName(pluginClass, true, null);
        }
        return (Class<Object>) getClass().getClassLoader().loadClass(pluginClass);
    } catch (ClassNotFoundException e) {
        throw new IllegalArgumentException("Plugin class not found " + pluginClass, e);
    }
}

I've never investigated why the bootstrap class loader could be null, since I had never faced to this problem

Graziano Felline

@ogipogi

Thank You Vladimir for the additional infos

Vladimir Dvorak

@skybber

If it fix your problem, we can add the fix to ha-core

Graziano Felline

@ogipogi

I will test this

Vladimir Dvorak

@skybber

thanks

I don't have JbossEAP at all ...

Graziano Felline

@ogipogi

:)
I have problems to build a local SNAPSHOT jar

Vladimir Dvorak

@skybber

OK, I'll prepare it

Graziano Felline

@ogipogi

Thank you very much

Vladimir Dvorak

@skybber

snapshot updated

Graziano Felline

@ogipogi

ok, I'll test it

Vladimir Dvorak

@skybber

btw, you need dcevm11 to build HA

Graziano Felline

@ogipogi

Ahh, that was maybe the issue

Vladimir Dvorak

@skybber

I'm using dcevm11 as system jdk now for a week

https://github.com/TravaOpenJDK/trava-jdk-11-dcevm-staged

Graziano Felline

@ogipogi

I tested it:

activemq is deployed
New exception during deployment of Camunda engine

14:46:08,971 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 69) MSC000001: Failed to start service jboss.deployment.unit."camunda-webapp-ee-jboss-7.11.11-ee.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."camunda-webapp-ee-jboss-7.11.11-ee.war".undertow-deployment: java.lang.NoClassDefFoundError: io/undertow/server/handlers/resource/ResourceManager
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.NoClassDefFoundError: io/undertow/server/handlers/resource/ResourceManager
    at java.lang.ClassLoader.findBootstrapClass(Native Method)
    at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1009)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:408)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:406)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.findSystemClass(ClassLoader.java:998)
    at org.jboss.modules.ConcurrentClassLoader.findSystemClassInternal(ConcurrentClassLoader.java:416)
    at org.jboss.modules.JDKSpecific.getSystemClass(JDKSpecific.java:180)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:395)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
    ... 8 more

I need to investigate what happened

Vladimir Dvorak

@skybber

You can try to disable UndertowPlugin

Graziano Felline

@ogipogi

I'm testing now exactly that :)

Vladimir Dvorak

@skybber

UndertowPlugin is for resource loading

Graziano Felline

@ogipogi

It looks good, the exception disappear.
Thank you very much Vladimir.
I'll test now the hot swapping.

Vladimir Dvorak

@skybber

OK, may be Undertow will be needed

then something is wrong in UndertowTransformer - PrefixingResourceManager is extended from jboss ResourceManager, that is not found at

at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)

Graziano Felline

@ogipogi

Ok, i get additional Exception during deployment of EAR file, also i need to investigate.

15:02:26,199 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.deployment.unit."app.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."app.ear".POST_MODULE: WFLYSRV0153: Failed to process phase POST_MODULE of deployment "app.ear"
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:151)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
    at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Error: java.lang.NullPointerException
    at org.jboss.as.weld.deployment.BeanDeploymentArchiveImpl.<init>(BeanDeploymentArchiveImpl.java:111)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.createExplicitBeanDeploymentArchive(BeanArchiveProcessor.java:347)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.processResourceRoot(BeanArchiveProcessor.java:304)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.handleResourceRoot(BeanArchiveProcessor.java:237)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor$ResourceRootHandler.access$100(BeanArchiveProcessor.java:206)
    at org.jboss.as.weld.deployment.processors.BeanArchiveProcessor.deploy(BeanArchiveProcessor.java:113)
    at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:144)
    ... 8 more
Caused by: java.lang.NullPointerException
    ... 15 more

15:02:26,213 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "app.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"app.ear\".POST_MODULE" => "WFLYSRV0153: Failed to process phase POST_MODULE of deployment \"app.ear\"
    Caused by: java.lang.Error: java.lang.NullPointerException
    Caused by: java.lang.NullPointerException"}}

During deployment of submodules i get now several new HA Info message from JBossModules plugin, this looks good.

HOTSWAP AGENT: 15:02:25.754 INFO (org.hotswap.agent.config.PluginRegistry) - Plugin 'org.hotswap.agent.plugin.jbossmodules.JBossModulesPlugin' initialized in ClassLoader 'ModuleClassLoader for Module "deployment.app.ear.my-module-a-impl.jar" from Service Module Loader'.

Graziano Felline

@ogipogi

Ok, this errors during deploy of EAR file are related with the last fix ( bootstrap class loader). If i deactivate Undertow and JBossModules i get the same Exception...

Vladimir Dvorak

@skybber

INFO messages are ok

bootstrap fix seem ok for me

have you rest of stack trace - pby there is part related to HA

you can try to disable WeldPlugin too

Graziano Felline

@ogipogi

ok, the exception disappear if i disable also the weld plugin

Vladimir Dvorak

@skybber

We have tests for wilfly up to 3.1.3
https://github.com/HotswapProjects/HotswapAgent/blob/master/plugin/hotswap-agent-weld-plugin/run-tests.sh#L79

But not in EAP

Graziano Felline

@ogipogi

ok, thank you for the info

Vladimir Dvorak

@skybber

not at all, hope it helps you

Graziano Felline

@ogipogi

:thumbsup:

Graziano Felline

@ogipogi

Ok, it looks lie the app i have is to large (large monolith).
The deployment of ear hangs, if i have JBossModules plugin activated.
It hangs on:

17:40:09,444 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment app.ear

jstack:

at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResource(ExternalBeanArchiveProcessor.java:330)
    at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResources(ExternalBeanArchiveProcessor.java:313)

With disabled JBossModules plugin deployment duration is ca. 8 min.

Any hints how i can patch JBossModules plugin for test if swapping works to an explicit module (ejb jar)?

Vladimir Dvorak

@skybber

look into ModuleClassLoaderTransformer

if you are not attached by external debugger, then you also need autoHotswap=true

There are generally 2 ways how the JVM can get new bytecode, 1. from debugger 2. from HA ussing autoHotswap

Vladimir Dvorak

@skybber

Anyway for Wildfly, I would try to remove unused plugins directly from hotswap-agent.jar. It will avoid a lot of useless scanning especially on jboss eap, where are a lot of classloaders that slow things down.

1 reply

I would recommend to test it outside docker at first, if it works, then docker is next level

Graziano Felline

@ogipogi

The reason for hanging (much slower deployment) is:

Setup has a lot of sub projects (large monolith)
I set hotswap-agent.properties in one of these sub projects src/main/resources folder withextraClasspath=[path_to_my_project]/build/classes/java/main
During deployment of other ejbs (other sub project) it checks now if on the extraClasspath it exists a META-INF/beans.xml or META-INF/jandex.idx file

Stack (java.nio.file.Files.exists(Files.java:2385))

java.lang.Thread.State: RUNNABLE
  at sun.nio.fs.UnixNativeDispatcher.access0(Native Method)
  at sun.nio.fs.UnixNativeDispatcher.access(UnixNativeDispatcher.java:449)
  at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:306)
  at java.nio.file.Files.exists(Files.java:2385)
  at org.jboss.modules.PathResourceLoader.lambda$getResource$4(PathResourceLoader.java:155)
  at org.jboss.modules.PathResourceLoader$$Lambda$7/2033524545.run(Unknown Source)
  at org.jboss.modules.PathResourceLoader.doPrivilegedIfNeeded(PathResourceLoader.java:248)
  at org.jboss.modules.PathResourceLoader.getResource(PathResourceLoader.java:155)
  at org.jboss.modules.ModuleClassLoader.loadResourceLocal(ModuleClassLoader.java:410)
  at org.jboss.modules.ModuleClassLoader$1.loadResourceLocal(ModuleClassLoader.java:144)
  at org.jboss.modules.Module.getResources(Module.java:861)
  at org.jboss.modules.Module.getExportedResources(Module.java:904)
  at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResource(ExternalBeanArchiveProcessor.java:330)
  at org.jboss.as.weld.deployment.processors.ExternalBeanArchiveProcessor.findExportedResources(ExternalBeanArchiveProcessor.java:313)

Is this expected?
Without hotswap-agent.properties file it doesn't hang anymore, but i have no chance to link to classes for swapping.

Vladimir Dvorak

@skybber

I didn't see that

Graziano Felline

@ogipogi

Hi Vladimir, which part don't you see?

Vladimir Dvorak

@skybber

I did not see the problem yet

its not clear what place of code calls the check procedure, it seems it is not related to HA

Graziano Felline

@ogipogi

It look like the defined extraClasspath has here an impact.
I need to understand this first - yes.
https://github.com/wildfly/wildfly/blob/eacb144910142d456946ea2c429f63abf0cf8b93/weld/subsystem/src/main/java/org/jboss/as/weld/deployment/processors/ExternalBeanArchiveProcessor.java#L309

Only to be sure - it is right, to place the hotswap-agent.propertiesfile in resources folder of subprojects?

Vladimir Dvorak

@skybber

yes, it is right place for project or module specific HA configuration

Where communities thrive

People

Repo info

Activity