Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 15:16
    gitoleg synchronize #1036
  • 13:25
    ivg assigned #1041
  • 13:14
    ivg assigned #1068
  • 13:14
    ivg opened #1068
  • 13:10
    ivg assigned #1067
  • 13:10
    ivg assigned #1067
  • 13:10
    ivg opened #1067
  • 13:09
    ivg assigned #1065
  • 13:09
    ivg assigned #1066
  • 13:09
    ivg opened #1066
  • 13:01
    ivg opened #1065
  • 12:58

    ivg on master

    updates testsuite (#1064) This… (compare)

  • 12:58
    ivg closed #1064
  • 12:58
    ivg closed #1063
  • Feb 26 21:49
    gitoleg opened #1064
  • Feb 26 21:02
    ivg opened #1063
  • Feb 26 21:00

    ivg on master

    fixes a bug in the primus reall… (compare)

  • Feb 26 21:00
    ivg closed #1062
  • Feb 26 20:59
    gitoleg opened #1062
  • Feb 26 15:54

    ivg on master

    adds a stub for realloc (#1035)… (compare)

Ivan Gotovchits
@ivg
@Auxy233 hope it helps :)
Ivan Gotovchits
@ivg
checked odoc's issue tracker, they indeed do not generate indices, its not our or odig's fault. The good news is that both search (ocaml/odoc#124) and indices (ocaml/odoc#127) are on their roadmap (ocaml/odoc#210).
Quoc Tran
@qqtranqq
in the logs for my crypto workload - i am getting a fair amount of unsupported instructions SHUFPS, AESENCLAST, AESKEYGENASSIST, MULXQ and ADOXQ
Ivan Gotovchits
@ivg
yep, we do not support all of the SSE et al instructions yet
Quoc Tran
@qqtranqq
is it hard to write my own bad support for those? I don't think I could do SHUFPS , but the XQ stuff doesn't seem terrible
but also i don't think unsupported instructions are the reason my workload is running slow - is there anything in particular I should be looking for in the logs?
Ivan Gotovchits
@ivg
@qqtranqq well, yeah unsupported instructions wouldn't slow down Primus) But supported would, and we do support a few complex x86 instructions.
First of all let's sync on the idea of "running low". Just to give you a figure, on average it takes slightly more than a second to investigate one path that is 4096 primitives long. The number of paths processed so far is in the primus-greedy output. The number of paths is roughly 2 to 10 times more than the number of basic blocks (this is basically the cyclomatic complexity of the program). Usually, we discover a new block in less than one second (again on average). So processing 50k blocks, which is roughly the number of blocks in a rather big program (1-5 Megabytes) might take up to 12hours.
Quoc Tran
@qqtranqq
Ok - I don't really mind certain analyses taking 12 hours for certain workloads - is there an intuition on how BAP scales if I was able to throw more hardware at it?
Ivan Gotovchits
@ivg
I think that your case is slower mostly due to the CISC nature of SSE instructions, and the coverage should also be quite bad, as we will run out of gas (exceed the allowed number of millinstructions per path) very soon. So my first idea would be switching to a more risc architecture, like if you have the same file (or can compile the same file) into ARM or MIPS or PPC then you might have more luck. (However, our support of special crypotinstructions is good for those architectures, but, at least, I hope it will be easier to add them for MIPS and PPC, as their lifters are much more modern and easier to work with).
Ivan Gotovchits
@ivg

is there an intuition on how BAP scales if I was able to throw more hardware at it?

We run several analysis on different binaries in parallel. The main bottleneck is memory, roughly per each binary of interest, reserve 10 to 20 Gigabytes. So, if you have a bunch of servers each having 128 Gb of RAM, you can run about 8 bap instances per server without trouble. This is a lower bound (i.e., I'm taking rather large binaries, such as firmwares with thousands of functions), in general, if your targets are smaller, i.e., a few dozens of kilobytes (like the average size of a program in /usr/bin, you can easily run 20+ instances per server.
Anyway, for us, the analyst time is the bottleneck and we're more focused on optimizing this time (i.e., to minimize the effort of an analyst, i.e., the between an incident report and CVE report) :)

Concerning adding support for those instructions, in the future, you should be able to write Primus Lisp stubs, but right now the only official way is to write a good old and very verbose semantics in OCaml. This is so tedious that I wouldn't even suggest you do this. I am not that amoral)))) But, I remember we were doing a trick when we were adding floating-point support and instead of lifting some complex floating-point instructions we were translating them into calls (i.e., like intrinsics) which we later stub with Primus Lisp... so we can employ the same approach, let me think if I can bring anything from that branch to the real-world)
Ivan Gotovchits
@ivg
besides, if you have the sources, you may also experiment with different architectural options and disable SSE, AVX, etc, as a workaround. But to summarize, yes, analyzing crypto code is extremely hard both for a human and machine(
Ivan Gotovchits
@ivg

@qqtranqq, see #1059, now we can translate any unknown instruction into a call to an intrinsic function, and then stub this function in Primus Lisp. In the future, when BAP Lisp will be ready, it should be possible to reify those stubs into static IR, but right now it will work for Primus. I think I will stub tomorrow floating-point operations (via OCaml floating-point operations) and see how it works.

The fun part, is that you can now disable lifters with (--no-x86 for example) and disassemble the whole program as a set of intrinsic calls (not sure why one would do this, besides the sacred "because we can"). In addition, we can now try evaluate programs for which we don't have lifters.

JT Paasch
@jtpaasch

How does Primus determine a valid stack pointer for x86_64 programs?

I noticed that if I run a simple program and set RSP to, say 0x0 or 0x7FFFFFFFFFFF, then when Primus encounters an instruction like RSP := RSP - 8, the program crashes/segfaults.

But if RSP is a value like 0x40000000, the program doesn't crash.

Ivan Gotovchits
@ivg
@jtpaasch, Well, the same as a real operating system loader. You can specify any base and any size (with --primus-loader-stack-base and --primus-loader-stack-size, which default to 0x40000000 and 8Mb correspondingly). Then it will map this region of memory, that's all.
A segfault happens when you're trying to access a memory region which is not mapped or which doesn't have the necessary permissions.
See plugins/primus_loader_basic.mli and --primus-loader-help in general, for more information and details.
JT Paasch
@jtpaasch
Amazing. Thanks @ivg!
Ivan Gotovchits
@ivg

@qqtranqq, with #1059 we can now stub missing instructions with Primus Lisp. But, in general, an ability to stub instructions opens a wide range of opportunities for us, that will allow us to tackle with complex binaries that do crypto (SSE/AVX) and floating-points. Both of them suffer from code explosion (because they use extremely complex instructions commonly millicoded in CPU's firmware), that (a) very hard to lift (they correspond to hundreds lines of code in C, which will take even more in BIL) and (b) hard to analyze. However, from the security perspective, we do not really need to go into these instructions and unfold their implementation, we just need to compute them concretely and as fast as possible without bloating our analysis space. So, the plan is to stub those instructions and compute them using the host CPU, which will be much faster and will relieve Primus from doing the hard work (and increase coverage, as basically right now every such complex instruction is an impassable barrier for Primus - it runs out of gas before it reaches even the end of the instruction).

Right now, as a proof of concept, I implemented some floating-point operations (well, they are also useful for analysis too), so we can easily compute programs with floating points. I hope this will help you (and it already helps us).

larkwiot
@larkwiot
Hello everyone, I am having trouble install bap from opam. lwt 5.1.1 is failing and links to a known github issue that says 5.0.0 made some minor breaking changes. Is there a way to fix or workaround this?
Ivan Gotovchits
@ivg
Try installing explicitly a lower version of lwt, e.g., opam install lwt.4.0.0 bap.2.0.0
Also, lwt is not an important dependency, so even if it fails, bap should work, only baptop won't be available
larkwiot
@larkwiot
Thanks! That worked
larkwiot
@larkwiot
In trying to build the jmp example from the main BAP github page, I cannot make it work. bapbuild doesn't seem to do anything when I do bapbuild jmp.ml in the same directory as the file. It says 0 targets found. Then bapbundle seems to work (if returning nothing is working) but the pass does not exist for bap, so I think it did not work. What am I doing wrong?
Ivan Gotovchits
@ivg
@larkwiot, you should do bapbuild jmo.plugin not ml ;)
bapbuild jmp.plugin
larkwiot
@larkwiot
Thank you!
Only problem now is that the copy/pasted example doesn't compile and errors with:
File "jmp.ml", line 16, characters 3-30:
Error: This expression has type unit but an expression was expected of type
(unit, error) result
Ivan Gotovchits
@ivg
Oops, @larkwiot, indeed I didn't update the example, here is the correct version (just add Ok ()), e.g.,
open Core_kernel
open Bap_main
open Bap.Std

let counter = object
  inherit [int * int] Term.visitor
  method! enter_term _ _ (jmps,total) = jmps,total+1
  method! enter_jmp _ (jmps,total) = jmps+1,total
end

let main proj =
  let jmps,total = counter#run (Project.program proj) (0,0) in
  printf "ratio = %d/%d = %g\n" jmps total (float jmps /. float total)

let () = Extension.declare @@ fun _ctxt ->
  Project.register_pass' main;
  Ok ()
I will update the README
larkwiot
@larkwiot
Thanks for the quick reply!
Excellent! It worked
Ivan Gotovchits
@ivg
Great!) Now you are ready for the next lesson ;)
https://github.com/BinaryAnalysisPlatform/bap-tutorial
larkwiot
@larkwiot
Oooo I didn't notice that before, thanks!
larkwiot
@larkwiot
So I'm seeing that when running bap on a given binary it is only hitting one core and barely touching my memory. Is there a way to either make it utilize more of my resources, or is that up to how I write the plugin, or what do I need to learn about bap's architecture to understand how it runs and uses resources?
Anton Kochkov
@XVilka
By the way, maybe next year you could apply to Google Summer of Code @ivg
Enkelmann
@Enkelmann
The bap-tutorial may need an overhaul for BAP 2.0. For example, bap --list-formats doesn't work anymore.
Ivan Gotovchits
@ivg
@Enkelmann, yeah, I am afraid to touch it, as I will start overhauling too much)
Ivan Gotovchits
@ivg
Ok, so the tutorial is updated a little bit (ideally we should also update to use the new command-line interface, it is much easier to grok)
@larkwiot, bap is currently single-threaded as OCaml is a single-threaded language. But if it weren't, utilizing several CPU for program analysis is an open question, as many (if not all) most demanding algorithms in program analysis are either impossible or hard to parallelize. Mostly because all algorithms are codependent. That is not to say that your cores should remain unused. We have many servers featuring lots of cpus running bap all fully loaded. We just run bap in parallel on different binaries :)
Ivan Gotovchits
@ivg
@XVilka, that's a good idea. However, we need the list of ideas the same as you have in radare2, before that. This is the most time-consuming part(
also, we do not have that much mentoring power, though if you will volunteer to mentor for BAP we can at least double it)
Ivan Gotovchits
@ivg

I've merged #1059 and #1061 into the master. Now Primus is roughly three times faster and we can turn instructions into intrinsic calls. First of all, this is useful for instructions with unknown semantics (so that we can easily and fastly stub their semantics in Primus Lisp, e.g., for example, it takes only 6 lines of code in Primus Lisp to stub SSE floating-point addition). Stubbing unknown instructions in Primus Lisp not only enables them for the emulation mode but also, in the future, it should be possible to translate those stubs into the BIL/IR static representation. This would be possible when we will finish our work on BAP Lisp (planned in BAP 2.2, which will be in October/September, this year). So right now you can take, say, an s390x binary, and do bap ./exe --no-ida --bil-enable-intrinsic=:unknown --run and gradually add semantics to the missing intrinsics.

This feature is also useful for various VEX instructions that are implemented as hardware millicode and unfold to tons of BIL code on which our analysis chokes (like many crypto instructions, CRC, and so on). One of the approaches is instead of modeling those instructions we can turn them into uninterpreted functions (and in Primus we can give those functions a concrete interpretation, the way we did with floating points).

Anton Kochkov
@XVilka
@ivg first time organizations usually get only one student anyway, so it is fine
Quoc Tran
@qqtranqq
I am excited for #1059 and #1061 in master!
the instruction stubbing for unknown instructions also doesn't look terrible
Quoc Tran
@qqtranqq
in #1035
(defun realloc/update-chunk (old-ptr new-len)
  (let ((old-len (malloc/get-chunk-size old-ptr)))
    (if (>= old-len len) (realloc/shrink-chunk ptr len)
      (let ((new-ptr (malloc new-len)))
        (when new-ptr
          (memcpy new-ptr old-ptr old-len)
          (free old-ptr))
        new-ptr))))
(realloc/shrink-chunk ptr len) is supposed to be (realloc/shrink-chunk old-ptr new-len) ?
gitoleg
@gitoleg
Looks like you're right, my bad. will fix it very soon!
Quoc Tran
@qqtranqq
when primus outputs "primus-lisp.info> skipping a jump to an unknown destination at 0xBEEF" does that mean it encountered an indirect jump?
Ivan Gotovchits
@ivg
no, it means that Primus has to jump to the given address, but this address is not available (either because it was missed by a disassembler, or, more often, because it is just a random number that resulted from concretization)
Ivan Gotovchits
@ivg
@qqtranqq, the issue from #1035 is resolved with #1062 and #1063 will prevent this from happening in the future :)