Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Fuyang Liu
    @liufuyang
    If you could come up with some simple code that involves some simple malloc and free operation with javacpp then I would be happy to help you test whether it works with ZGC in my IDE.
    Otherwise if you are very sure about this issue cannot possibly originated via javacpp then I will report this to tensorflow java again.
    Fuyang Liu
    @liufuyang
    It would feel weird that some tensorflow-java code is not correct that can make a ZGC flag not working properly? 🤔 Anyway I know not much in tf-java nor javacpp it's just intuitively it feels the error is not rooted from there.
    Samuel Audet
    @saudet
    JavaCPP comes with unit tests that check that stuff out and I've just verified that everything runs fine with _JAVA_OPTIONS="-XX:+UnlockExperimentalVMOptions -XX:+UseZGC" mvn clean install and OpenJDK 11.
    I do think it's a problem with TF Java. Please read this thread, you may find it enlightening: deepjavalibrary/djl#690
    Fuyang Liu
    @liufuyang
    Thanks. I will try read the thread later. If your unit test all runs at very small scale then perhaps you cannot verify a memory leak issue?
    With the test I did the memory usage went up to 1G after around 50000 cycles of allocation.
    But for us as long as we remove the ZGC option it all works for us. So I am only reporting this here in the hope of ironing out some potential issues around the libraries which might having memory leaks when ZGC is used. I will not have more time and energy to help much with further debug or getting this issue eventually solved.
    Fuyang Liu
    @liufuyang
    My colleague in another team wasted a week before he realised it was the ZGC that is causing this issue. So anyway I hope my post of the issue can help anyone who encounter the same problem to firstly try turn off ZGC and see what happens.
    Samuel Audet
    @saudet
    Well, like I said, it's not going to do anything with GC when we set "org.bytedeco.javacpp.nopointergc" to "true". So, just set that and you're all set.
    And that's great that you were able to debug this, but it's not a good idea to rely on GC anyway. It cannot be relied upon.
    Issues like that will continue to happen, until you simply stop to try and do things with GC!
    It's never ever always going to work correctly. You will encounter issues like that over and over again.
    Fuyang Liu
    @liufuyang
    "until you simply stop to try and do things with GC"
    Hmm.... I get very confused when you say things like this
    I mean... If I am writing a Java application, the GC is always there, isn't it?
    And as I have mentioned that, setting this org.bytedeco.javacpp.nopointergc=true has no effect, the ZGC flag still gives error.
    Maybe we have misunderstanding of what ZGC flag does? I think it only replace a Java GC algorithm, from the default G1GC to something else.
    It is not we wanted to turn on some GC in our application. It was just because ZGC seemed to be a better GC than the default GC so we used that all the time until we had this out of memory bug.
    Fuyang Liu
    @liufuyang

    And that's great that you were able to debug this, but it's not a good idea to rely on GC anyway. It cannot be relied upon.

    I don't understand this. In Java there is always GC, I cannot turn it off and we were not rely on anything. -XX:+UseZGC is simply telling Java "use a different (experimenting but hopefully better) GC policy".

    Fuyang Liu
    @liufuyang
    Sorry I will have to leave this conversation now. I have posted the issue on tf java as well. It seems we don't have a directly answer why this happens. Probably not many people are using ZGC anyway so this is not some major issue anyway. It's only when people are using ZGC and encounter this issue I hope it would be easier for them to find out that it might be related to ZGC.
    Samuel Audet
    @saudet
    @liufuyang I'm talking about GC in relation to JavaCPP and native code in general. It was never designed to be used to manage native resources, and should never be used for anything critical. If you're having issues with GC and JavaCPP, we can disable it by setting the "org.bytedeco.javacpp.nopointergc" system property to "true". If you're still having issues even when doing this, then the issue is probably unrelated to JavaCPP. So, thank you for posting this tensorflow/java#315, and let's continue the discussion there.
    Brad Hards
    @bradh
    Has anyone seen presets that will sometimes produce java.lang.NoClassDefFoundError exceptions? I see it as Could not initialize class org.bytedeco.ffmpeg.global.swscale, and random suggestions are usually around static initialisers. It mostly works for me locally, and dies in the github CI during test, which is ugly to debug. Any suggestions appreciated. Will see what I can find in any case.
    Samuel Audet
    @saudet
    Make sure that the "org.bytedeco.javacpp.logger.debug" system property is set to "true" to get more information on the console
    Brad Hards
    @bradh
    As in mvn ... -Dorg.bytedeco.javacpp.logger.debug=true ?
    Samuel Audet
    @saudet
    Yes, that works if your code runs as part of Maven's JVM, which isn't the case of Surefire...
    Brad Hards
    @bradh
    Thanks for the reminder. Presumably Failsafe too.
    Debug: Loading /home/runner/.javacpp/cache/ffmpeg-4.3.1-1.5.4-linux-x86_64.jar/org/bytedeco/ffmpeg/linux-x86_64/libswscale.so.5
    Debug: Creating symbolic link /home/runner/.javacpp/cache/ffmpeg-4.3.1-1.5.4-linux-x86_64.jar/org/bytedeco/ffmpeg/linux-x86_64/libswscale.so
    Debug: Locking /home/runner/.javacpp/cache before extracting
    Debug: Failed to extract for jniswscale: java.lang.UnsatisfiedLinkError: java.nio.channels.FileLockInterruptionException
    Brad Hards
    @bradh
    Is that likely to be from the preset or from javacpp?
    Samuel Audet
    @saudet
    That's something that's calling Thread.interrupt(), that's not from JavaCPP or the presets.
    Brad Hards
    @bradh
    I was trying to understand what might be making that message. Looks like https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Loader.java#L1758
    Why its sometimes getting interrupt()ed is perhaps a race between that loading sequence and the calling code?
    doktor-ziel
    @doktor-ziel

    Hello:)
    I saw that the issue like

    [ERROR] Failed to execute goal org.bytedeco:javacpp:1.5.6-SNAPSHOT:parse (javacpp-parser) on project cocoa: Failed to execute JavaCPP Builder: /home/doktor/code/javacpp-presets/cocoa/cppbuild/linux-x86_64/include/CoCoA-0.99712/TmpMorseElement.H:551:Unexpected token 'EOF' ->

    Occurred several times to couple of you. I saw one suggestion to build the presets on Centos here. However, when I tried I got an error:

    autoreconf: running: /usr/bin/autoconf --force
    autoreconf: configure.ac: not using Autoheader
    autoreconf: running: automake --add-missing --copy --force-missing
    configure.ac:15: error: require Automake 1.14.1, but have 1.13.4
    autoreconf: automake failed with exit status: 1

    I also saw that the EOF error can be solved otherway (like bytedeco/javacpp-presets#889). However, the parsed file is quite simple in point of view of C++ macros...

    What should I do to be added to contributors of a project? I'd be able to push my feature branch to share more details about my issue.

    Samuel Audet
    @saudet
    Start by going through https://github.com/bytedeco/javacpp/wiki/Mapping-Recipes, but if you already have something, feel free to show what you have, we can help from there. Thanks!
    doktor-ziel
    @doktor-ziel
    I've read that page several times:)
    Can you let me to do push to the presets repository?
    doktor-ziel
    @doktor-ziel
    Parser has problem with this file
    Samuel Audet
    @saudet
    You can create your fork and we can look at it, sure.
    doktor-ziel
    @doktor-ziel
    Good
    doktor-ziel
    @doktor-ziel

    I forked your repo and introduced my changes to symbollic-algebra branch: https://github.com/doktor-ziel/javacpp-presets/tree/symbollic-algebra

    It would be great if somebody looks at it:)

    Samuel Audet
    @saudet
    Instead of including all the header files in the presets all at once, start with only a single header file, and get that working, first, then add a second header file, get that working next, and repeat that for all the rest.
    Jason Lee
    @jasondlee
    i'm working through the adding a preset instructions, and my build is failing because it can't find the header file to include. the native library builds, but the java part fails. how/where is that configured?
    and hi :)
    Samuel Audet
    @saudet
    Hi, if you're using the Maven plugin, that's the includePaths here in the pom.xml file:
    https://github.com/bytedeco/javacpp-presets/blob/master/pom.xml#L188
    Jason Lee
    @jasondlee
    i am. thanks! :)
    i've made some pretty decent progress today. :) failing now trying to compile the generated JNI classes, iiuc. i'll get there :)
    Brad Hards
    @bradh
    As part of trying to upgrade to the new GPL/LPGL ffmpeg preset, I'm seeing something weird with the licenses. Tools like cyclonedx seem to think the LGPL version is GPL. I see something similar at https://mvnrepository.com/artifact/org.bytedeco/ffmpeg/4.3.2-1.5.5
    Not sure where it is getting the license info from - doesn't seem to be in the .pom.
    Samuel Audet
    @saudet
    Brad Hards
    @bradh
    Thanks. Will see if it works to override it in a child pom.xml