Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 09:29
    tobim commented #556
  • Aug 21 15:15
    tobim labeled #556
  • Aug 21 15:15
    tobim labeled #556
  • Aug 21 15:15
    tobim opened #556
  • Aug 21 14:38

    tobim on ch7947

    Integrate CAF into cmake build (compare)

  • Aug 20 19:02

    mavam on warnings

    (compare)

  • Aug 20 19:02

    mavam on master

    Fix gcc9 warnings Merge pull request #555 Fix gc… (compare)

  • Aug 20 19:02
    mavam closed #555
  • Aug 20 14:01

    tobim on ch7947

    Add caf and broker as submodules (compare)

  • Aug 20 08:12
    tobim synchronize #555
  • Aug 20 08:12

    tobim on warnings

    Fix gcc9 warnings (compare)

  • Aug 19 15:43
    tobim review_requested #555
  • Aug 19 15:43
    tobim opened #555
  • Aug 19 15:43
    tobim labeled #555
  • Aug 19 15:41

    tobim on warnings

    Fix gcc9 warnings (compare)

  • Aug 19 13:31

    mavam on ch7642

    (compare)

  • Aug 19 13:31

    mavam on master

    Fix printing of conjunctions C… Restrict PCAP expressions to pa… Add CHANGELOG entry and 1 more (compare)

  • Aug 19 13:31
    mavam closed #554
  • Aug 18 10:27
    mavam review_request_removed #554
  • Aug 18 10:27
    mavam review_requested #554
JasperEm
@JasperEm
Ok until then I will improve the code quality
Matthias Vallentin
@mavam
:+1:
Joseph Noir
@josephnoir
I build VAST with the clang 6 from homebrew and there are three tests failing: command, streambuf und option_map. Any recommendation with commit to use for a somewhat stable build?
Dominik Charousset
@Neverlord
command and option_map fail because of new features, so they shouldn't affect using VAST. streambuf is only broken on Linux, we'll look into it.
Joseph Noir
@josephnoir
Thx, did something change with the import porcess? When I try to import data, this happens (using the master branch):
$ cat ~/vast/libvast/test/logs/bro/conn.log | vast import bro
     _   _____   __________
    | | / / _ | / __/_  __/
    | |/ / __ |_\ \  / /
    |___/_/ |_/___/ /_/  0.1

~/vast/libvast/vast/system/reader_command.hpp:62: assertion failed 'input'
~/vast/build/lib/libvast.so.0(_ZN4vast6system14reader_commandINS_6format3bro6readerEE11make_sourceERN3caf12scoped_actorERKNS_10option_mapENSt3__111__wrap_iterIPKNSC_12basic_stringIcNSC_11char_traitsIcEENSC_9allocatorIcEEEEEESM_+0x832)[0x7ff3534573c2]
~/vast/build/lib/libvast.so.0(_ZN4vast6system19reader_command_base8run_implERN3caf12actor_systemERKNS_10option_mapENSt3__111__wrap_iterIPKNS8_12basic_stringIcNS8_11char_traitsIcEENS8_9allocatorIcEEEEEESI_+0x4c)[0x7ff35357e86c]
~/vast/build/lib/libvast.so.0(_ZN4vast7command3runERN3caf12actor_systemERNS_10option_mapENSt3__111__wrap_iterIPKNS6_12basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEESG_+0x29d)[0x7ff35329079d]
~/vast/build/lib/libvast.so.0(_ZN4vast7command3runERN3caf12actor_systemERNS_10option_mapENSt3__111__wrap_iterIPKNS6_12basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEESG_+0x385)[0x7ff353290885]
~/vast/build/lib/libvast.so.0(_ZN4vast7command3runERN3caf12actor_systemERNS_10option_mapENSt3__111__wrap_iterIPKNS6_12basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEESG_+0x385)[0x7ff353290885]
~/vast/build/lib/libvast.so.0(_ZN4vast7command3runERN3caf12actor_systemENSt3__111__wrap_iterIPKNS4_12basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEEEESE_+0x28)[0x7ff353291448]
vast(main+0x6c)[0x400f2c]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ff351333445]
vast[0x400de9]
Aborted (core dumped)
Matthias Vallentin
@mavam
Fixed in master, @josephnoir. There was an inconsistency in the option declaration for the user input.
Joseph Noir
@josephnoir
:+1:
Joseph Noir
@josephnoir
Should VAST currently build with the CAF master branch?
Dominik Charousset
@Neverlord
Yes, it does for me. What error do you get?
Joseph Noir
@josephnoir
$ make
[  1%] Building CXX object libvast/CMakeFiles/libvast.dir/src/bitmap.cpp.o
[  1%] Building CXX object libvast/CMakeFiles/libvast.dir/src/compression.cpp.o
[  2%] Building CXX object libvast/CMakeFiles/libvast.dir/src/concept/hashable/crc.cpp.o
[  3%] Building CXX object libvast/CMakeFiles/libvast.dir/src/concept/hashable/xxhash.cpp.o
[  3%] Building CXX object libvast/CMakeFiles/libvast.dir/src/data.cpp.o
[  4%] Building CXX object libvast/CMakeFiles/libvast.dir/src/detail/adjust_resource_consumption.cpp.o
[  4%] Building CXX object libvast/CMakeFiles/libvast.dir/src/detail/compressedbuf.cpp.o
In file included from /Users/noir/Code/bmbf-demo/vast/libvast/src/bitmap.cpp:14:
In file included from /Users/noir/Code/bmbf-demo/vast/libvast/vast/bitmap.hpp:16:
In file included from /Users/noir/Code/bmbf-demo/actor-framework/libcaf_core/caf/variant.hpp:25:
/Users/noir/Code/bmbf-demo/actor-framework/libcaf_core/caf/default_sum_type_access.hpp:64:14: error: no member named 'get_data' in 'vast::bitmap'
    return x.get_data().template apply<Result>(std::forward<Visitor>(visitor),
           ~ ^
/Users/noir/Code/bmbf-demo/actor-framework/libcaf_core/caf/sum_type.hpp:137:28: note: in instantiation of function template specialization 'caf::default_sum_type_access<vast::bitmap>::apply<void, caf::visit_impl_continuation<void, 0, (lambda at /Users/noir/Code/bmbf-demo/vast/libvast/src/bitmap.cpp:62:18) &> &>' requested here
    return trait::template apply<Result>(x, continuation,
                           ^
/Users/noir/Code/bmbf-demo/actor-framework/libcaf_core/caf/sum_type.hpp:166:49: note: in instantiation of function template specialization 'caf::visit_impl<void, 1>::apply<(lambda at /Users/noir/Code/bmbf-demo/vast/libvast/src/bitmap.cpp:62:18) &, const vast::bitmap &>' requested here
  return visit_impl<Result, sizeof...(Ts) + 1>::apply(std::forward<Visitor>(f),
                                                ^
/Users/noir/Code/bmbf-demo/vast/libvast/src/bitmap.cpp:68:8: note: in instantiation of function template specialization 'caf::visit<(lambda at /Users/noir/Code/bmbf-demo/vast/libvast/src/bitmap.cpp:62:18) &, const vast::bitmap &, void>' requested here
  caf::visit(visitor, bm);
       ^
1 error generated.
make[3]: *** [libvast/CMakeFiles/libvast.dir/src/bitmap.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [libvast/CMakeFiles/libvast.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [all] Error 2
Joseph Noir
@josephnoir
(Using the clang 6.0 installed via homebrew.)
Matthias Vallentin
@mavam
Ah, sorry about that. The reason for this bug: we had some internal miscommunication and #193 got merged before it was ready. I'll fix this in a bit.
Joseph Noir
@josephnoir
What is a rough time frame for a bit, today, this week, ...?
Matthias Vallentin
@mavam
In a few hours. If master doesn't compile, that's a big flashing red light!
Joseph Noir
@josephnoir
K, thx! :+1:
Matthias Vallentin
@mavam
We're still behind with our Jenkins deployement, which we have only deployed internally. This shouldn't have happened in the first place...
Dominik Charousset
@Neverlord
My bad. The PR looked like a straightforward thing to me. Didn't notice the remaining todos in the PR.
Matthias Vallentin
@mavam
This stuff happens from time to time, but hey, thankfully the damage is marginal in this case.
Matthias Vallentin
@mavam
@josephnoir Should be fixed now.
Joseph Noir
@josephnoir
:smile:
Joseph Noir
@josephnoir
Ok, got another question, the IP in PREFIX syntax seems to fail for me when importing MRT files:
$ gunzip -c bgp/updates.20180530.23*.gz | vast import mrt "114.215.46.175 in prefix || MORE SIMILAR QUERIES"
level = trace, node = 55421E6B34240EBFE340A36970C1BCA6AFF68B00#46584
     _   _____   __________
    | | / / _ | / __/_  __/
    | |/ / __ |_\ \  / /
    |___/_/ |_/___/ /_/  0.1

mrt-reader sets filter expression to: prefix ni 114.215.46.175
connect to remote node: direct
connecting to 127.0.0.1 : 42000
got node
signal-monitor sends signals to 6@55421E6B34240EBFE340A36970C1BCA6AFF68B00#46584
signal-monitor registers signal handler for Hangup: 1
signal-monitor registers signal handler for Interrupt: 2
signal-monitor registers signal handler for Quit: 3
signal-monitor registers signal handler for Terminated: 15
signal-monitor registers signal handler for User defined signal 1: 30
signal-monitor registers signal handler for User defined signal 2: 31
connecting source to importers
mrt-reader registers sink 14@55421E6B34240EBFE340A36970C1BCA6AFF68B00#46428
/Users/noir/Code/bmbf-demo/vast/libvast/src/expression.cpp:138: assertion failed '!caf::holds_alternative<none>(*x)'
0   libvast.0.dylib                     0x00000001000ac024 _ZN4vast6tailorERKNS_10expressionERKNS_4typeE + 532
1   libvast.0.dylib                     0x0000000100293a92 _ZZN4vast6system6sourceINS_6format3mrt6readerEEEN3caf8behaviorEPNS5_14stateful_actorINS0_12source_stateIT_EENS5_17event_based_actorEEEOS9_ENKUlNS5_13atom_constantILNS5_10atom_valueE4161203EEEE_clESH_ + 1218
2   libvast.0.dylib                     0x0000000100293558 _ZN3caf18trivial_match_caseIZN4vast6system6sourceINS1_6format3mrt6readerEEENS_8behaviorEPNS_14stateful_actorINS2_12source_stateIT_EENS_17event_based_actorEEEOSA_EUlNS_13atom_constantILNS_10atom_valueE4161203EEEE_E6invokeERNS_6detail21invoke_result_visitorERNS_17type_erased_tupleE + 184
3   libcaf_core.0.15.7.dylib            0x0000000100e12c25 _ZN3caf6detail13behavior_impl6invokeERNS0_21invoke_result_visitorERNS_17type_erased_tupleE + 69
4   libcaf_core.0.15.7.dylib            0x0000000100e69478 _ZN3caf15scheduled_actor7consumeERNS_15mailbox_elementE + 2232
5   libcaf_core.0.15.7.dylib            0x0000000100e64710 _ZN3caf15scheduled_actor10reactivateERNS_15mailbox_elementE + 32
6   libcaf_core.0.15.7.dylib            0x0000000100e6e796 _ZN3caf9intrusive16drr_cached_queueINS_6policy15urgent_messagesEE9new_roundINS0_28w
But, I also get an error when trying a similar filter with the bro conn log. The error looks different though:
$  cat ../../vast/libvast/test/logs/bro/conn.log | vast import bro "id.orig_h in 192.168.0.0"
level = trace, node = 55421E6B34240EBFE340A36970C1BCA6AFF68B00#46653
...
connect to remote node: direct
connecting to 127.0.0.1 : 42000
bro-reader sets filter expression to: id.orig_h in 192.168.0.0
got node
signal-monitor sends signals to 6@55421E6B34240EBFE340A36970C1BCA6AFF68B00#46653
signal-monitor registers signal handler for Hangup: 1
signal-monitor registers signal handler for Interrupt: 2
signal-monitor registers signal handler for Quit: 3
signal-monitor registers signal handler for Terminated: 15
signal-monitor registers signal handler for User defined signal 1: 30
signal-monitor registers signal handler for User defined signal 2: 31
connecting source to importers
bro-reader registers sink 14@55421E6B34240EBFE340A36970C1BCA6AFF68B00#46428
bro-reader parsed bro header:
bro-reader     #separator     
bro-reader     #set_separator ,
bro-reader     #empty_field (empty)
bro-reader     #unset_field -
bro-reader     #path conn
bro-reader     #fields:
bro-reader       0 ) ts : time
bro-reader       1 ) uid : string
bro-reader       2 ) id.orig_h : addr
bro-reader       3 ) id.orig_p : port
bro-reader       4 ) id.resp_h : addr
bro-reader       5 ) id.resp_p : port
bro-reader       6 ) proto : string
bro-reader       7 ) service : string
bro-reader       8 ) duration : duration
bro-reader       9 ) orig_bytes : count
/Users/noir/Code/bmbf-demo/vast/libvast/vast/system/source.hpp:114: assertion failed 'x'
bro-reader       10 ) resp_bytes : count
bro-reader       11 ) conn_state : string
bro-reader       12 ) local_orig : bool
bro-reader       13 ) missed_bytes : count
bro-reader       14 ) history : string
bro-reader       15 ) orig_pkts : count
bro-reader       16 ) orig_ip_bytes : count
bro-reader       17 ) resp_pkts : count
bro-reader       18 ) resp_ip_bytes : count
bro-reader       19 ) tunnel_parents : set<string>
bro-reader auto-detected field 0 as event timestamp
0   libvast.0.dylib                     0x000000010b7c976c _ZZN4vast6system6sourceINS_6format3bro6readerEEEN3caf8behaviorEPNS5_14stateful_actorINS0_12source_stateIT_EENS5_17event_based_actorEEEOS9_ENKUlNS5_13atom_constantILNS5_10atom_valueE4161203EEEE_clESH_ + 3308
1   libvast.0.dylib                     0x000000010b7c8a08 _ZN3caf18trivial_match_caseIZN4vast6system6sourceINS1_6format3bro6readerEEENS_8behaviorEPNS_14stateful_actorINS2_12source_stateIT_EENS_17event_based_actorEEEOSA_EUlNS_13atom_constantILNS_10atom_valueE4161203EEEE_E6invokeERNS_6detail21invoke_result_visitorERNS_17type_erased_tupleE + 184
2   libcaf_core.0.15.7.dylib            0x000000010c387c25 _ZN3caf6detail13behavior_impl6invokeERNS0_21invoke_result_visitorERNS_17type_erased_tupleE + 69
3   libcaf_core.0.15.7.dylib            0x000000010c3de478 _ZN3caf15scheduled_actor7consumeERNS_15mailbox_elementE + 2232
4   libcaf_core.0.15.7.dylib            0x000000010c3d9710 _ZN3caf15scheduled_actor10reactivateERNS_15mailbox_elementE + 32
5   libcaf_core.0.15.7.dylib            0x000000010c3e3796 _ZN3caf9intrusive16drr_cached_queueINS_6policy15urgent_messagesEE9new_roundINS0_28wdrr_fixed_multiplexed_queueINS2_11categorizedENS1_INS2_15normal_messagesEEEJNS0_9drr_queueINS2_17upstream_messagesEEENS0_30wdrr_dynamic_multiplexed_queueINS2_19downstream_messagesEEES4_EE26new_round_recursion_helperILm3ES4_NS_15scheduled_actor15mailbox_visitorEEEEENS0_16new_round_resultEmRT_ + 166
6   libcaf_core.0.15.7.dylib            0x000000010c3daf74 _ZN3caf9intrusive10fifo_inboxINS_15scheduled_actor14mailbox_policyEE9new_roundINS2_15mailbox_visitorEEENS0_16new_round_resultEmRT_ + 516
7   libcaf_core.0.15.7.dylib            0x000000010c3d9f35 _ZN3caf15scheduled_actor6resumeEPNS_14execution_unitEm + 485
8   libcaf_core.0.15.7.dylib            0x000000010c36507d _ZN3caf9scheduler6workerINS_6policy13work_stealingEE3runEv + 205
9   libcaf_core
Not sure if these are connected or it is a coincidence that it fails with both queries.
Matthias Vallentin
@mavam
We'll look into it. Can't promise it today, but I hope to have a fix as of tomorrow.
Joseph Noir
@josephnoir
Oh, the latter error was my mistake, wrong types I guess. Adding a /16 makes it work. (cat ../../vast/libvast/test/logs/bro/conn.log | vast import bro "id.orig_h in 192.168.0.0/16").
Matthias Vallentin
@mavam
Oh, good.
Admittedly, there should be a reasonable error message and not a segfault. :unamused:
Joseph Noir
@josephnoir
That would be nice!
Joseph Noir
@josephnoir
The problem with MRT persists and I'm somewhat sure that the types are right and names match. If you need access to the data let me know.
Matthias Vallentin
@mavam
Okay, I'll get back to you if I can't find the cause elsewhere.
Joseph Noir
@josephnoir
Did you have time to look at the bug yet?
Matthias Vallentin
@mavam
Not quite yet, sorry.
Joseph Noir
@josephnoir
K, no problem. I’ll try to use my old branch then. Was hoping to get rid of some of the bugs, but I have to start setting things up for a demo. :smile:
Joseph Noir
@josephnoir
Did I say no problem? Looks like my branch does not work. :worried: I want to say it did, but I'm probably wrong about that. Because why wouldn't it work anymore. So ... I tried a few different CAF commints and a different compiler, but no luck. Also checked out the earliest merge of the MRT branch into the master and the filtered import of MRT data with the prefix expression fails as well. For whatever reason importing the data and than doing and export with the same filter works fine. Any pointers to how I coul debug this? (This bug also happens if I start a continues query and then export the MRT data without a filter, but the main VAST instance crashes, not the source or sink.)
Dominik Charousset
@Neverlord
Let's chat on site about this tomorrow.
Joseph Noir
@josephnoir
K
GTrunSec
@GTrunSec
How to find specified logs such as conn or dns. This is command
vast -e localhost:42000 export -e 10 ascii '&name == "bro::conn"'
I have tried Example command vast -e localhost:42000 export bro &name == "bro::conn".
Does the &name is deprecated on master branch?
GTrunSec
@GTrunSec
Did the &name change to expr = to<expression>("&type == bro::conn");
Matthias Vallentin
@mavam
Right, it's deprecated. The &name extractor has been replaced with &type.
I'm sorry that the documentation is lagging behind.
We're preparing the 0.1 release for January, which will come with improved documentation and a reasonable quick start tutorial.
GTrunSec
@GTrunSec
awesome! I'm looking forward new tutorial. But I still got a problem vast -e localhost:42000 export bro '&type == "bro::conn"' vast.ec(unspecified, ("component already exists"))
Matthias Vallentin
@mavam

How did you start VAST? You don't need to add the -e switch unless you run a non-default setting. We recommend running in the client-server mode, where you have two processes:

  • vast start gets the server up; this process runs continuously

And then to ingest/query, there will be short-running processes:

  • vast import ... to send data to the server
  • vast export ... to get data from the server
GTrunSec
@GTrunSec
  • vast -e localhost:42000 start for server up
  • cat ~/src/test/conn.log | vast -e localhost:42000 import bro send log to server
  • vast -e localhost:42000 export -e 10 ascii :addr in 10.0.0.0/8 I got output
  • vast -e localhost:42000 export bro '&type == "bro::conn"' So, I just want know how to search specified logs. Maybe I am using wrong symbols of command
Matthias Vallentin
@mavam

It looks correct, but you're not getting any results?

For convenience, I would rewrite the commands like this:

  • vast start (defaults to localhost:42000)
  • vast import bro < ~/src/test/conn.log
  • vast export -e 10 ascii ':addr in 10.0.0.0/8'
  • vast export ascii '&type == "bro::conn"'
Actually, try the ascii output for the last query. Does that make a difference?
GTrunSec
@GTrunSec
I got it. Thank you for helping
Matthias Vallentin
@mavam
Good to hear!