Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Danny van Bruggen
@matozoid
Maybe, like me, you ignored it? ;-)
edefazio
@edefazio
yeah... I guess my tech radar is a little rusty
Danny van Bruggen
@matozoid
@albert_monfa_twitter - you can see some [TESTCASE] PR's here, if you can create one of those it would be great.
Danny van Bruggen
@matozoid
JDK 14 is already in rampdown :-O
Better get the new language features in JavaParser quickly
Danny van Bruggen
@matozoid
edefazio
@edefazio
nice
Chinese News Board
@ralphcnb_twitter
Does anybody know about "Memory Heap" or "Memory Consumption" of the Java-Parser? As I have said, I have written an HTML-Parse Library that is getting better & better. I have added a "Tool" that uses my HTML-Parse-Search which Parser the "JavaDoc Tool" output that is bundled with the JDK. About a month ago, I started using the "Java-Parser Library" to pull out Method-Bodies (as Strings), and also the Field-Delcarations (as Strings), and then use an external Code-HiLiter to insert "HiLited Method Bodies" (along with other, user-defined, stuff) into Java-Doc Generated Documentation Pages. My HTML Parse Classes are extremely light-weight. I do not build "Parse-Trees" at all, but rather Lists (Vectors, to be precise). The JAVA GARBAGE COLLECTOR has never had any problems cleaning my Memory Heap of stale Vector<HTMLNode>. HOWEVER, with my Java-Doc-Upgrader Tool, I have lots of "Parse-Trees" in Memory... AND The Java-Garbage-Collection Routines don't seem to be solving any of it... Does anybody know anything about "Cleaning the Heap" after a parse?
Chinese News Board
@ralphcnb_twitter
Also, this is the HTML-ized "Log File" from a typical "Java-Doc-Upgrader Tool" Run. I added a Verbose-Memory-Print so that all the text that is in purple is how many Mega-Bytes the tool is consuming on the Heap... I execute Java's Runtime.gc() after "Upgrading" each HTML Java-Doc Web-page. The Garbage-Collector usually "Free's Memory" for my Vector<HTMLNode>, basically because - as I said - I don't build any "Parse Trees" (nor DOM-Tree's) when parsing HTML. I treat HTML like a "List", but I know that the AST from Java-Parser is a tree, and therefore has million pointers, and therefore is probably more difficult for the Garbage-Collector to clean... Here is the link of my last "Build"
By the end of it there are 200 MegaBytes of "stuff" on the Heap that won't go away. I cannot see how they could be HTMLNode Vector's, they must be StaticJavaParser Node's.
edefazio
@edefazio
I'm going to take a wild guess that the problem is, you retain at reference to one or more nodes (i.e. the JavadocComments) instead of the Strings representing the comment/nodes...the reference wont GC because of the way JP has bidirectional references... (in JavadocComments case, a reference to the Node it is commenting) If you have a reference to ANY node within the AST in JavaParser (i.e. even the smallest IntegerLiteralExpr) it has a reference to it's parent, and so on and so forth. if the Javadoc comment never goes out of scope, you'll keep the whole thing in memory
edefazio
@edefazio

i.e.

package a.b.c;
public class C{
    /** I am a javadoc comment*/ 
    int i = 2;

}

...if I read in the above Java class... and ONLY extract /use the JavadocComment instance /* I am a javadoc comment /
this comment has a pointer reference to it's commented entity:
(in Comment base class)

    @InternalProperty
    private Node commentedNode;

and that Node (a FieldDeclaration int i=2; ) will have a reference to it's parent : ClassOrInterfaceDeclaration
and that node will have a reference to it's parent, the CompilationUnit

Chinese News Board
@ralphcnb_twitter
The "Wild Guess" was INCORRET, Se├▒or. I do not maintain ANY pointers to the Native-Java-Parser Libraries Abstrast-Syntax-Trees, nor Parse-Trees. Earlier on, when I started a month ago, I was doing that, but I have since cut that out, and left it as an "option." My Question, can be boiled down to this: "I am using the class/method combination com.github.javaparser.StaticJavaParser.parse(String s), and also the class/method com.github.javaparser.StaticJavaParser.parseMethodDeclaration(String methodDeclaration), is there any change that the class "StaticJavaParser" keeps back-pointers or reference-pointers to the Abstract-Syntax-Trees that it is building?" Let me know... I am getting about 300 MB of "stuff" that the Java-Garbage-Collector will not seem to clean. NOTE: I am quite illiterate in the "ways and means" of the implementation of Java JDK "Garbage Collection Process." ALSO-NOTE: I have good reason for this, it has always "taken out the trash" of my (ultra-light) Data-Classes without any problems at all. This is why I have never worried about the Java System.gc()..... (I also know that AST's have many, many internal pointers, which is the EXACT-OPPOSITE of what my Vector<HTMLNode>'s are... - but, then again, Vectorized-HTML and AST's are fundamentally different data-structures...)
edefazio
@edefazio
well... you are just a joy to work with... I'm glad I just spent 30 mins looking over your code trying to figure out where you were using JavaParser...
It only has the parser configuration, and that has no links to compilation units or their contents.
StaticJavaParser really is just a utility to more easily access the JavaParser class itself. Every static methods makes an instance of JavaParser and forgets about it after parsing.
Now, if you use the symbol solver, well, that is known to be memory intensive, and there are several issues examining that.
Chinese News Board
@ralphcnb_twitter
OK... Well... I don't know. I have not "worried" to much about memory - not in this way I took it "head on" by only creating ultra-light-weight data classes for my Vector<HTMLNode>. That's my project. Actually, my "original project" was downloading and translating News-Websites largely from Asia, but it eventually included Spanish-Pages too, since I have studied both many times in my life, and lived on both of those places before (Taipei 1.5 years, Mexico City 2.5 years)....
Here is the "Core Java-Parser" of my "Java-Parser Bridge" Classes... I guess I will try to post the code:
 public static final boolean keepNativeAPIReference = false;
 Method (MethodDeclaration md)
    {
        NodeList<Modifier>      ml  = md.getModifiers();
        NodeList<Parameter>     pl  = md.getParameters();
        NodeList<ReferenceType> te  = md.getThrownExceptions();

        this.nativeJP_API_reference = keepNativeAPIReference ? md : null;
        this.name                   = md.getNameAsString();
        this.signature              = md.getDeclarationAsString();
        this.returnType             = md.getType().toString();
        this.returnFullType         = keepNativeAPIReference ? md.getType() : null;
        this.body                   // = md.getBody().isPresent() ? md.getBody().get().toString() : null;  // In ConstructorDeclaration, this is an Optional<BlockStmt>, not here!
                                    // = md.getBody().isPresent()
                                    //   ? SourceFile.prettyPrinter.print(md.getBody().get().toString())
                                    //   : null;
                                    = md.getBody().isPresent()
                                        ?   com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.print(
                                            com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.setup(md.getBody().get()))
                                        :   null;
        this.modifiers              = new String[ml.size()];
        this.parameterNames         = new String[pl.size()];
        this.parameterTypes         = new String[pl.size()];
        this.parameterFullTypes     = keepNativeAPIReference ? new com.github.javaparser.ast.type.Type[pl.size()] : null;
        this.exceptions             = new String[te.size()];
        this.jdComment              = md.hasJavaDocComment() ? md.getJavadocComment().get().toString() : null;

        int i = 0;
        for (Parameter p : pl)
        {
            parameterNames[i]       = p.getName().toString();
            parameterTypes[i]       = p.getType().toString();

            // This will eat up more memory on the heap, and is completely unnecessary for the Java-Doc Tool to run.
            // However, a user might wish to make use of this, so it is left to the public-boolean "keepNativeAPIReference"
            if (keepNativeAPIReference) parameterFullTypes[i]   = p.getType();
            i++;
        }
        i = 0;
        for (Modifier m : ml)       modifiers[i++]  = m.toString();
        i = 0;
        for (ReferenceType r : te)  exceptions[i++] = r.toString();
    }
Danny van Bruggen
@matozoid
Looks alright.
You can have a go with this tool: https://visualvm.github.io/
It can show you in detail what's in memory.
(I wouldn't worry about 200MB though. Maybe the JVM thinks it is not worth cleaning this stuff up or so)
Chinese News Board
@ralphcnb_twitter
The lines below is "The Value" that made "incorporating method bodies into JavaDoc Generated HTML Pages' a million times easier:
 this.body    = md.getBody().isPresent()
                                  ?   com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.print(
                                       com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.setup(md.getBody().get()))
                                   :   null;
// This line is used to transmit it to "http://HiLite.me" and provide HTML-ified HiLited Java-Source Code.
// Afterwards, that HTML is inserted into each and every Method-Details Section of each and every Class, Inteface & Enumerated-Type
// Of a Java-Doc Page.  This part works, but like I said, this is creating a lot of extra-memory use on the heap
// It's not a "MAJOR PROBLEM" .. but it does "SLOW THE JRE DOWN" eventually...
Danny van Bruggen
@matozoid
Are you trying to get the exact formatting of the method body there? Interesting approach.
Danny van Bruggen
@matozoid
    public static void main(String[] args) {
        CompilationUnit cu = StaticJavaParser.parse("class X{void x(){}}");
        System.out.println(cu.findFirst(MethodDeclaration.class)
                .map(md -> md.getTokenRange().get().toString())
                .orElse("??"));
    }
This is a bit of a hidden feature: the tokens that the parser used are available from the Nodes. They are the exact sequence of low level syntactic text pieces that form the source code, including whitespace, comments, everything. So if you take the tokens that form the method declaration and print them in sequence, you get the source code back.
Chinese News Board
@ralphcnb_twitter
It all works just fine. No problems. The "Stringified" method is retrieved fine, and the HiLite.Me server hilites the method. My Vectorised HTML works too. The heap grows, and the sad/funny thing is that when Runtime.gc() is invoked, as the heap grows - if the GC fails to clean the heap, the GC gets slower and slower and slower...
dkkacceler
@dkkacceler
I have a question, javaparser is able to handle the (spring boot)@autowired annotation´╝č
Suppose I have two ServiceImpl1,2, when I use @autowired TestService testservice, how does javaparser know which is the correct impl?
edefazio
@edefazio
@dkkacceler depends on how you are using JavaParser, but JavaParser doesn't use (take as input) the Runtime Impl (the ServiceImpl1,2) but rather the .java source for some code... Spring @autowired is processed at runtime by annotation processing
so effectively the fields on the "Spring Bean" that is @autowired will not be set (all the @autowired stuff doesn't affect the .java source code of a Class, but rather the runtime/in memory Object instance... to JavaParser, .java source is source, and there is only ONE version of ServiceImpl.
edefazio
@edefazio
I have a more detailed discussion related to this (based on Lombok, which uses Compile time annotations) here #2436
dkkacceler
@dkkacceler
Thank you for help
dkkacceler
@dkkacceler
Does javaparser provide APIs for query interface implementation classes?If any, I can iterate through the impls and find the serviceimpl I want
Danny van Bruggen
@matozoid
Not sure, but I assume you have to look at every class and ask it for what it extends.
edefazio
@edefazio

I think the issue here is that :
JavaParser runs at "source time" with input as a String (representing .java source code)

Spring operates as an annotation processor at Runtime (mostly) and by this time we are talking in Java bytecode
(Thats why Spring uses tools like ASM to manipulate the bytecode)
https://www.baeldung.com/java-asm

In short you have two separate "lifecycles" operating on separate "code formats"
(Strings/Tokens/Nodes... for JavaParser during source-time...i.e. like manipulating code in your IDE)
(Bytecodes...for Spring & ASM... doing bytecode manipulation at runtime during startup/annotation processing)

So the changes made to the bytecode by Spring (or the runtime objects/instances) will be operating on runtime bytecode and not the source Strings

If... you are building something akin to a Pluggable Service Locator (i.e. I have the interface, and I want to "resolve" some implementation based on all the available implementations in my classpath) you might want to use a Tool like : https://github.com/classgraph/classgraph
edefazio
@edefazio
FWIW I am aware that Spring doesn't always have to "change" the bytecode (in the case of Simple dependency injection) but I'm just trying to point out that, by the time Spring is running, it is going to be operating with Runtime instances & bytecode ...and not be "reading/manipulating source code" which is JavaParser's domain
Chinese News Board
@ralphcnb_twitter
Speaking of Code-Injection ... If you guys don't know much about retained references in StaticJavaParser (because I parse around 500 Java Source Code Files in my build - for making better code-documentation that now has "Inline Code Hiliting" - and it helps to 'take out the trash' after I am done with a file...).
Chinese News Board
@ralphcnb_twitter
Speaking of Code Injection, would it be difficult to help me, sort of, "Fix a JDK Bug" ... Which I would want to do by "injecting an accessor / getter method into a certain generics class in the java.util.* library? I need access to a field named modCount in the AbstractList or the AbstractCollection class, so I can implement my Iterators better, and throw an exception that I really want to be able to throw, but cannot because I cannot access the field in the standard JDK implementation of these generics... Anybody know? Is it possible at Run-Time to insert a getter into standard-JDK class AbstractLust for protected field modCount, in "real-time" or "at run time" ??? I guarantee you that it's a bonafide bug in the JDK not to allow some kind of accessor method for this field... I need to be able to throw something known as "ConcurrentModificationException" in the specialized Iterators that I have written... But I cannot properly check for the Situations without access to the field - unless I horribly inefficient version of "the checker" ... Let me know...
Xx
Chinese News Board
@ralphcnb_twitter

I would want to "Insert" this getter method into a the JDK Class AbstractList

public int getModeCount() { return modCount; }

// So that I would be able to check in my "Iterators"

{
    if (getModCount() != expectedModCount) throw new
        ConcurrentModificationException("Underlying Vector Modified outside of ListIterator, this is not allowed.");
    ...
   // Rest of Iterator Code
}

I would have to do this to the "Java JDK Byte Code for AbstractList" after the class had been loaded into memory by the ClassLoader. I cannot re-implement my own inherited version of AbstractList and simply add this method, because it is of much higher priority to allow programmer to reuse class java.util.Vector<E> and never force them to use my own "proprietary" Collections or List. That's half the value of the package. Is this possible?

Danny van Bruggen
@matozoid
Jan Ouwens, from the company where I work, had a pretty popular talk for a while about messing around with the JVM. https://jqno.nl/talks/dont-hack-the-platform/
Java may not be the language for you though. There's not a language around that is less hacker friendly!