Better get the new language features in JavaParser quickly
Does anybody know about "Memory Heap" or "Memory Consumption" of the Java-Parser? As I have said, I have written an HTML-Parse Library that is getting better & better. I have added a "Tool" that uses my HTML-Parse-Search which Parser the "JavaDoc Tool" output that is bundled with the JDK. About a month ago, I started using the "Java-Parser Library" to pull out Method-Bodies (as Strings), and also the Field-Delcarations (as Strings), and then use an external Code-HiLiter to insert "HiLited Method Bodies" (along with other, user-defined, stuff) into Java-Doc Generated Documentation Pages. My HTML Parse Classes are extremely light-weight. I do not build "Parse-Trees" at all, but rather Lists (Vectors, to be precise). The JAVA GARBAGE COLLECTOR has never had any problems cleaning my Memory Heap of stale Vector<HTMLNode>. HOWEVER, with my Java-Doc-Upgrader Tool, I have lots of "Parse-Trees" in Memory... AND The Java-Garbage-Collection Routines don't seem to be solving any of it... Does anybody know anything about "Cleaning the Heap" after a parse?
Here is the whole package (again): http://developer.torello.directory/JavaHTML/Version%201/1.5/javadoc/index.html
Also, this is the HTML-ized "Log File" from a typical "Java-Doc-Upgrader Tool" Run. I added a Verbose-Memory-Print so that all the text that is in purple is how many Mega-Bytes the tool is consuming on the Heap... I execute Java's Runtime.gc() after "Upgrading" each HTML Java-Doc Web-page. The Garbage-Collector usually "Free's Memory" for my Vector<HTMLNode>, basically because - as I said - I don't build any "Parse Trees" (nor DOM-Tree's) when parsing HTML. I treat HTML like a "List", but I know that the AST from Java-Parser is a tree, and therefore has million pointers, and therefore is probably more difficult for the Garbage-Collector to clean... Here is the link of my last "Build"
By the end of it there are 200 MegaBytes of "stuff" on the Heap that won't go away. I cannot see how they could be HTMLNode Vector's, they must be StaticJavaParser Node's.
I'm going to take a wild guess that the problem is, you retain at reference to one or more nodes (i.e. the
JavadocComments) instead of the Strings representing the comment/nodes...the reference wont GC because of the way JP has bidirectional references... (in JavadocComments case, a reference to the Node it is commenting) If you have a reference to ANY node within the AST in JavaParser (i.e. even the smallest IntegerLiteralExpr) it has a reference to it's parent, and so on and so forth. if the Javadoc comment never goes out of scope, you'll keep the whole thing in memory
i.e.
package a.b.c;
public class C{
/** I am a javadoc comment*/
int i = 2;
}...if I read in the above Java class... and ONLY extract /use the JavadocComment instance /* I am a javadoc comment /
this comment has a pointer reference to it's commented entity:
(in Comment base class)
@InternalProperty
private Node commentedNode;and that Node (a FieldDeclaration int i=2; ) will have a reference to it's parent : ClassOrInterfaceDeclaration
and that node will have a reference to it's parent, the CompilationUnit
The "Wild Guess" was INCORRET, Señor. I do not maintain ANY pointers to the Native-Java-Parser Libraries Abstrast-Syntax-Trees, nor Parse-Trees. Earlier on, when I started a month ago, I was doing that, but I have since cut that out, and left it as an "option." My Question, can be boiled down to this: "I am using the class/method combination com.github.javaparser.StaticJavaParser.parse(String s), and also the class/method com.github.javaparser.StaticJavaParser.parseMethodDeclaration(String methodDeclaration), is there any change that the class "StaticJavaParser" keeps back-pointers or reference-pointers to the Abstract-Syntax-Trees that it is building?" Let me know... I am getting about 300 MB of "stuff" that the Java-Garbage-Collector will not seem to clean. NOTE: I am quite illiterate in the "ways and means" of the implementation of Java JDK "Garbage Collection Process." ALSO-NOTE: I have good reason for this, it has always "taken out the trash" of my (ultra-light) Data-Classes without any problems at all. This is why I have never worried about the Java System.gc()..... (I also know that AST's have many, many internal pointers, which is the EXACT-OPPOSITE of what my Vector<HTMLNode>'s are... - but, then again, Vectorized-HTML and AST's are fundamentally different data-structures...)
It only has the parser configuration, and that has no links to compilation units or their contents.
StaticJavaParser really is just a utility to more easily access the JavaParser class itself. Every static methods makes an instance of JavaParser and forgets about it after parsing.
Now, if you use the symbol solver, well, that is known to be memory intensive, and there are several issues examining that.
OK... Well... I don't know. I have not "worried" to much about memory - not in this way I took it "head on" by only creating ultra-light-weight data classes for my Vector<HTMLNode>. That's my project. Actually, my "original project" was downloading and translating News-Websites largely from Asia, but it eventually included Spanish-Pages too, since I have studied both many times in my life, and lived on both of those places before (Taipei 1.5 years, Mexico City 2.5 years)....
Here is the "Core Java-Parser" of my "Java-Parser Bridge" Classes... I guess I will try to post the code:
public static final boolean keepNativeAPIReference = false;
Method (MethodDeclaration md)
{
NodeList<Modifier> ml = md.getModifiers();
NodeList<Parameter> pl = md.getParameters();
NodeList<ReferenceType> te = md.getThrownExceptions();
this.nativeJP_API_reference = keepNativeAPIReference ? md : null;
this.name = md.getNameAsString();
this.signature = md.getDeclarationAsString();
this.returnType = md.getType().toString();
this.returnFullType = keepNativeAPIReference ? md.getType() : null;
this.body // = md.getBody().isPresent() ? md.getBody().get().toString() : null; // In ConstructorDeclaration, this is an Optional<BlockStmt>, not here!
// = md.getBody().isPresent()
// ? SourceFile.prettyPrinter.print(md.getBody().get().toString())
// : null;
= md.getBody().isPresent()
? com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.print(
com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.setup(md.getBody().get()))
: null;
this.modifiers = new String[ml.size()];
this.parameterNames = new String[pl.size()];
this.parameterTypes = new String[pl.size()];
this.parameterFullTypes = keepNativeAPIReference ? new com.github.javaparser.ast.type.Type[pl.size()] : null;
this.exceptions = new String[te.size()];
this.jdComment = md.hasJavaDocComment() ? md.getJavadocComment().get().toString() : null;
int i = 0;
for (Parameter p : pl)
{
parameterNames[i] = p.getName().toString();
parameterTypes[i] = p.getType().toString();
// This will eat up more memory on the heap, and is completely unnecessary for the Java-Doc Tool to run.
// However, a user might wish to make use of this, so it is left to the public-boolean "keepNativeAPIReference"
if (keepNativeAPIReference) parameterFullTypes[i] = p.getType();
i++;
}
i = 0;
for (Modifier m : ml) modifiers[i++] = m.toString();
i = 0;
for (ReferenceType r : te) exceptions[i++] = r.toString();
}
You can have a go with this tool: https://visualvm.github.io/
It can show you in detail what's in memory.
(I wouldn't worry about 200MB though. Maybe the JVM thinks it is not worth cleaning this stuff up or so)
The lines below is "The Value" that made "incorporating method bodies into JavaDoc Generated HTML Pages' a million times easier:
this.body = md.getBody().isPresent()
? com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.print(
com.github.javaparser.printer.lexicalpreservation.LexicalPreservingPrinter.setup(md.getBody().get()))
: null;
// This line is used to transmit it to "http://HiLite.me" and provide HTML-ified HiLited Java-Source Code.
// Afterwards, that HTML is inserted into each and every Method-Details Section of each and every Class, Inteface & Enumerated-Type
// Of a Java-Doc Page. This part works, but like I said, this is creating a lot of extra-memory use on the heap
// It's not a "MAJOR PROBLEM" .. but it does "SLOW THE JRE DOWN" eventually...
This is a bit of a hidden feature: the tokens that the parser used are available from the
Nodes. They are the exact sequence of low level syntactic text pieces that form the source code, including whitespace, comments, everything. So if you take the tokens that form the method declaration and print them in sequence, you get the source code back.
It all works just fine. No problems. The "Stringified" method is retrieved fine, and the HiLite.Me server hilites the method. My Vectorised HTML works too. The heap grows, and the sad/funny thing is that when
Runtime.gc() is invoked, as the heap grows - if the GC fails to clean the heap, the GC gets slower and slower and slower...
Suppose I have two ServiceImpl1,2, when I use @autowired TestService testservice, how does javaparser know which is the correct impl?
so effectively the fields on the "Spring Bean" that is @autowired will not be set (all the @autowired stuff doesn't affect the .java source code of a Class, but rather the runtime/in memory Object instance... to JavaParser, .java source is source, and there is only ONE version of ServiceImpl.
I have a more detailed discussion related to this (based on Lombok, which uses Compile time annotations) here #2436
I think the issue here is that :
JavaParser runs at "source time" with input as a String (representing .java source code)
Spring operates as an annotation processor at Runtime (mostly) and by this time we are talking in Java bytecode
(Thats why Spring uses tools like ASM to manipulate the bytecode)
https://www.baeldung.com/java-asm
In short you have two separate "lifecycles" operating on separate "code formats"
(Strings/Tokens/Nodes... for JavaParser during source-time...i.e. like manipulating code in your IDE)
(Bytecodes...for Spring & ASM... doing bytecode manipulation at runtime during startup/annotation processing)
So the changes made to the bytecode by Spring (or the runtime objects/instances) will be operating on runtime bytecode and not the source Strings
If... you are building something akin to a Pluggable Service Locator (i.e. I have the interface, and I want to "resolve" some implementation based on all the available implementations in my classpath) you might want to use a Tool like : https://github.com/classgraph/classgraph
FWIW I am aware that Spring doesn't always have to "change" the bytecode (in the case of Simple dependency injection) but I'm just trying to point out that, by the time Spring is running, it is going to be operating with Runtime instances & bytecode ...and not be "reading/manipulating source code" which is JavaParser's domain
Speaking of Code-Injection ... If you guys don't know much about retained references in
StaticJavaParser (because I parse around 500 Java Source Code Files in my build - for making better code-documentation that now has "Inline Code Hiliting" - and it helps to 'take out the trash' after I am done with a file...).
Speaking of Code Injection, would it be difficult to help me, sort of, "Fix a JDK Bug" ... Which I would want to do by "injecting an accessor / getter method into a certain generics class in the
java.util.* library? I need access to a field named modCount in the AbstractList or the AbstractCollection class, so I can implement my Iterators better, and throw an exception that I really want to be able to throw, but cannot because I cannot access the field in the standard JDK implementation of these generics... Anybody know? Is it possible at Run-Time to insert a getter into standard-JDK class AbstractLust for protected field modCount, in "real-time" or "at run time" ??? I guarantee you that it's a bonafide bug in the JDK not to allow some kind of accessor method for this field... I need to be able to throw something known as "ConcurrentModificationException" in the specialized Iterators that I have written... But I cannot properly check for the Situations without access to the field - unless I horribly inefficient version of "the checker" ... Let me know...
Xx
I would want to "Insert" this getter method into a the JDK Class AbstractList
public int getModeCount() { return modCount; }
// So that I would be able to check in my "Iterators"
{
if (getModCount() != expectedModCount) throw new
ConcurrentModificationException("Underlying Vector Modified outside of ListIterator, this is not allowed.");
...
// Rest of Iterator Code
}I would have to do this to the "Java JDK Byte Code for AbstractList" after the class had been loaded into memory by the ClassLoader. I cannot re-implement my own inherited version of AbstractList and simply add this method, because it is of much higher priority to allow programmer to reuse class java.util.Vector<E> and never force them to use my own "proprietary" Collections or List. That's half the value of the package. Is this possible?
Jan Ouwens, from the company where I work, had a pretty popular talk for a while about messing around with the JVM. https://jqno.nl/talks/dont-hack-the-platform/
Java may not be the language for you though. There's not a language around that is less hacker friendly!