Hi everyone! I'm using python subprocess to write commands into the Joern interactive shell. As my dataset is quite large i decided to split the process into many subsets. This is working fine for a small number of splits but for >3 splits the Joern interactive shell freezes and doesn't process. Is there a limit of calls that can be processed after each other in the interactive shell or should this be possible without closing it in between the calls?
@rasmusli_gitlab if you could share the script, that would be great. We can base a long term solution on it.
@rasmusli_gitlab Hello, are you also reproducing the devign paper? I encountered the same problem as you during this process. After the revision, joern does not seem to support vertex node traversal, so that the "graph-for-funcs.sc" script does not run successfully. Do you have a solution to this problem now, and if yes, can you share it?Thank you very much.
2 replies
@fabsx00 What is the way to call Joern in Python to analyze the source code of string format, not to generate files in workspace? There are so many files generated in this way that it is very slow.
joern> importCode(inputPath="./x42/c", projectName="x42-c")
Creating project
Project with name x42-c already exists - overwriting
Support for this language is only available in ShiftLeft Ocular with an appropriate license
res0: Option[Cpg] = None
Creating project
x42-c for code at ./x42/cProject with name x42-c already exists - overwriting
Support for this language is only available in ShiftLeft Ocular with an appropriate license
res0: Option[Cpg] = None
I'm wondering what's the problem
@colorlight did you try downloading the latest version of Joern using the instructions at https://docs.joern.io/installation?
I've just tested the quickstart instructions on a Linux machine and they work as expected.
If that doesn't help, could you post your system details so we can look for potential issues with the distribution?
I've just tested the quickstart instructions on a Linux machine and they work as expected.
If that doesn't help, could you post your system details so we can look for potential issues with the distribution?
@xiaotianming If I understood you correctly, and you'd like to generate a CPG for a subset of files found in a project directory, then I suggest that you point Joern at the subdirectory you intend to analyze. You won't get around generated files in the workspace, that's part of Joern's core functionality. As to your question about Python, you can start Joern as a server (https://docs.joern.io/server) and use a Python client library (https://github.com/joernio/cpgqls-client-python) to send it commands
@DogeWatch ^^
@xiaotianming triggering an analysis has some ramp-up time, so if you trigger multiple on small inputs, they may end up costing more time than a single large one. If you want to generate a large amount of CPGs using joern, then you might have to set up your own data processing pipeline, maybe using custom scripts (https://docs.joern.io/interpreter)
Hi, How can i use export Joern CPG into (node.csv, edge.csv) or other file format which neo4j can read it can how can I export the three into python? I found that in old version of joern and neo4j, It is sure that change the data path of neo4j can do that, but in new version of joern or in new version of neo4j, that can't be done. So how can I export the CPG14 in neo4j and python?
according to "Modeling and Discovering Vulnerabilities with Code Property Graphs", control flow edges need to be labeled: "While these edges need not be ordered as in the
case of the abstract syntax trees, it is necessary to assign a label of true, false or ε to each edge." how can these labels be accessed in joern? i assumed that edge labels are modeled as edge properties. but i cannot find a single control flow edge with any properties
case of the abstract syntax trees, it is necessary to assign a label of true, false or ε to each edge." how can these labels be accessed in joern? i assumed that edge labels are modeled as edge properties. but i cannot find a single control flow edge with any properties
Hey guys, I installed joern today and encountered the same problem as @colorlight while trying the stuff from your documentation.
I am running a VM with Ubuntu 16.04.5
Thanks in advance for looking into it!
hey @FJuilia_twitter! Joern features a step named ddgIn you can use to follow data dependency edges. For example, in the following program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char *argv[]) {
if (argc > 1 && strcmp(argv[1], "42") == 0) {
fprintf(stderr, "It depends!\n");
exit(42);
}
printf("What is the meaning of life?\n");
exit(0);
}you can follow DDG edges for the call to strcmp like so:
joern> cpg.call.name("strcmp").ddgIn.l
res103: List[nodes.TrackingPoint] = List(
Literal(
id -> 1000117L,
code -> "0",
order -> 2,
argumentIndex -> 2,
typeFullName -> "int",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(6),
columnNumber -> Some(43),
depthFirstOrder -> None,
internalFlags -> None
),
MethodParameterIn(
id -> 1000104L,
code -> "char *argv[]",
order -> 2,
name -> "argv",
evaluationStrategy -> "BY_VALUE",
typeFullName -> "char * [ ]",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(5),
columnNumber -> Some(19)
)
)
Additionally,
reachableBy might also help:joern> cpg.call.name("strcmp").reachableBy(cpg.method.parameter).l
res105: List[MethodParameterIn] = List(
MethodParameterIn(
id -> 1000104L,
code -> "char *argv[]",
order -> 2,
name -> "argv",
evaluationStrategy -> "BY_VALUE",
typeFullName -> "char * [ ]",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(5),
columnNumber -> Some(19)
)
)
@ursachec thank you for your answer :) unfortunately, your solution does not seem to work. ddgIn always yields an empty list, including if i try your example. i also noticed that ddgOut does not exist:
joern> cpg.call.name("strcmp").ddgIn.l
res59: List[nodes.TrackingPoint] = List()
joern> cpg.call.name("strcmp").l
res60: List[Call] = List(
Call(
id -> 1000112L,
code -> "strcmp(argv[1], \"42\")",
name -> "strcmp",
order -> 1,
methodInstFullName -> None,
methodFullName -> "strcmp",
argumentIndex -> 1,
dispatchType -> "STATIC_DISPATCH",
signature -> "TODO assignment signature",
typeFullName -> "ANY",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(6),
columnNumber -> Some(18),
resolved -> None,
depthFirstOrder -> None,
internalFlags -> None
)
)
joern> cpg.call.name("strcmp").ddgIn.l
res61: List[nodes.TrackingPoint] = List()
joern> cpg.call.name("strcmp").ddgOut.l
cmd62.sc:1: value ddgOut is not a member of overflowdb.traversal.Traversal[io.shiftleft.codepropertygraph.generated.nodes.Call]
val res62 = cpg.call.name("strcmp").ddgOut.l
^
Compilation Failedif i try reachableBy, i also just get an empty list:
joern> cpg.call.name("strcmp").reachableBy(cpg.method.parameter).l
res62: List[MethodParameterIn] = List()is there a command that needs to be called first so that these commands work? like a command to build the DDG?
@ursachec thank you, it works now! :) however, i cannot re-create the data flow example of the paper "Modeling and Discovering Vulnerabilities with Code Property Graphs". the paper contains PDG
for this code:
void foo()
{
int x = source();
if (x < MAX)
{
int y = 2 * x;
sink(y);
}
}i'm trying to prove using joern that there is data flow between int x = source() and sink(y). via ./joern-export --repr ddg --out outdir i get output that includes:
"1000105" -> "1000109" [ label = "x"]
"1000102" -> "1000109"
"1000116" -> "1000114" [ label = "2"]
"1000116" -> "1000114" [ label = "x"]
"1000102" -> "1000114"
"1000102" -> "1000116"
"1000109" -> "1000116" [ label = "x"]
"1000114" -> "1000119" [ label = "y"]from this, i can see that 1000105 → 1000109 → 1000116 → 1000114 → 1000119 is a path. 1000105 is int x = source() and 1000119 is sink(y). this proves the data flow. now i want to re-create this in joern. because ddgOut does not seem to exist, i'm walking backwards (starting at the sink): https://pastebin.com/U8VHFBWD i eventually get to 1000106L which is the call to source() but i never get to the assignment call int x = source()
reachableBy() does not seem to be the solution because that does not find the data flow either:
joern> cpg.call.id(1000105L).reachableBy(cpg.call.id(1000105L)).l
res100: List[Call] = List(
Call(
id -> 1000105L,
code -> "x = source()",
name -> "<operator>.assignment",
order -> 2,
methodInstFullName -> None,
methodFullName -> "<operator>.assignment",
argumentIndex -> 2,
dispatchType -> "STATIC_DISPATCH",
signature -> "TODO assignment signature",
typeFullName -> "ANY",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(3),
columnNumber -> Some(6),
resolved -> None,
depthFirstOrder -> None,
internalFlags -> None
)
)
joern> cpg.call.id(1000119L).reachableBy(cpg.call.id(1000119L)).l
res101: List[Call] = List(
Call(
id -> 1000119L,
code -> "sink(y)",
name -> "sink",
order -> 3,
methodInstFullName -> None,
methodFullName -> "sink",
argumentIndex -> 3,
dispatchType -> "STATIC_DISPATCH",
signature -> "TODO assignment signature",
typeFullName -> "ANY",
dynamicTypeHintFullName -> List(),
lineNumber -> Some(7),
columnNumber -> Some(3),
resolved -> None,
depthFirstOrder -> None,
internalFlags -> None
)
)
joern> cpg.call.id(1000105L).reachableBy(cpg.call.id(1000119L)).l
res102: List[Call] = List()
joern> cpg.call.id(1000119L).reachableBy(cpg.call.id(1000105L)).l
res103: List[Call] = List()i printed the reachability of the nodes to themselves first so you can be sure that i'm at the correct nodes. do you know why this doesn't work? :)
@FJuilia_twitter I am not 100% certain, but I think the behavior you're seeing is because when you're referencing the call to the assigment operator, you're actually referring to the return value of that call, which in your case, is not part of the flow.
if you'd take the source as being the identifier
if you'd take the source as being the identifier
x at line 3, you'd find a flow, and similarly for the call to source also at line 3
def source = cpg.identifier.lineNumber(3)
def sink = cpg.call.name("sink")
sink.reachableBy(source).l
@MeNicefellow it might be a better idea to export to the dot format: https://docs.joern.io/exporting/
Anyone got any idea how to get the line number corresponds to a cpg node?
When I use joern-export to export it to dot files.
@FJuilia_twitter it depends what you mean by gets written to. For assignments like
x = 2, you can search the graph for CALL nodes with the assignment operator as their method, e.g. cpg.call.methodFullName(Operators.assignment).l. If you're looking for byte-copying stdlib functions with a specific variable as argument, you would search for cpg.call.code(".*strcpy.*").where(_.argument.codeExact("x")) . Other steps from the reference card might be helpful https://docs.joern.io/cpgql/reference-card