Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 09 08:38
    adulau commented #267
  • May 09 08:38
    adulau closed #266
  • May 09 08:38

    adulau on main

    chg: [warninglists] Restructure… Merge pull request #267 from Wa… (compare)

  • May 09 08:38
    adulau closed #267
  • May 08 18:29
    Wachizungu opened #267
  • May 05 09:13
    Wachizungu edited #245
  • May 05 06:05
    adulau labeled #266
  • May 04 22:23
    Wachizungu opened #266
  • May 04 16:25
    adulau commented #265
  • May 04 16:25

    adulau on main

    chg: [User stories] Fix typo Merge pull request #265 from Wa… (compare)

  • May 04 16:25
    adulau closed #265
  • May 04 09:24
    Wachizungu opened #265
  • May 01 21:01
    Wachizungu commented #264
  • May 01 20:42
    adulau commented #264
  • May 01 20:42

    adulau on main

    chg: [FAQ] add sudo with user w… Merge pull request #264 from Wa… (compare)

  • May 01 20:42
    adulau closed #264
  • May 01 18:41
    Wachizungu opened #264
  • Apr 24 13:07

    adulau on main

    chg: [automation] add doc for /… Merge pull request #263 from Wa… (compare)

  • Apr 24 13:07
    adulau closed #263
  • Apr 24 12:10
    Wachizungu opened #263
Chris Lott
@chrisInMtown_twitter
none of our feeds processed by supplied MISP/Cake/delta ingest use object-attributes. Only our custom feeds; and MISP/Cake cannot fetch/cache those
andras
@andras:matrix.circl.lu
[m]
It should be able to cache those too
Unless i completely missed the point
Chris Lott
@chrisInMtown_twitter
that's why I say "MISP/Cake cannot cache"
The problem is the ETL task of consuming the premium data feed and munging it into events and attributes;
andras
@andras:matrix.circl.lu
[m]
Yeah depending on the format that might need a fair bit of glue
Chris Lott
@chrisInMtown_twitter
hmm questioning my assumptions here .. I believe that "cache" is an action that copies from the source (the remote server) to the local redis cache.
andras
@andras:matrix.circl.lu
[m]
It’s even dumber than that
Chris Lott
@chrisInMtown_twitter
If however "cache" can operate as copying from local MISP database of events, obj & attributes to local Redis memcache, then that would be very interesting
andras
@andras:matrix.circl.lu
[m]
It loops the remote through the ingestion that you would normally use for a fetch operation
Extracts the values from the derived attributes
Hashes them
And throws them into Redis
When it comes to misp format feeds/misp servers
It does something different
Chris Lott
@chrisInMtown_twitter
Thanks for the details. but for our in-house-developed ingestion scripts that copy remote to local database, I don't know how to get that data into the redis cache/analysis chain
andras
@andras:matrix.circl.lu
[m]
What format are they in?
Chris Lott
@chrisInMtown_twitter
some of our in-house ingestion scripts create object attributes
oh boy, format. Some XML, some JSON
andras
@andras:matrix.circl.lu
[m]
That should be fine
If you convert them to misp format already
You’re half way there
Generate a misp feed out of them
Chris Lott
@chrisInMtown_twitter
hmm starting to question everything we're doing :/
andras
@andras:matrix.circl.lu
[m]
Hehe
Chris Lott
@chrisInMtown_twitter
we use Python & PyMISP to massage the data and inject straight to events & attributes & objects
andras
@andras:matrix.circl.lu
[m]
So basically what we do normally
If we have a source in a format misp can’t cope with
Chris Lott
@chrisInMtown_twitter
I think you're suggesting, if we consume a premium feed and transform it to a MISP-friendly format, then we could use the MISP ingester, which would also allow us to use the MISP cacher. Did I understand correctly?
andras
@andras:matrix.circl.lu
[m]
We generate a misp feed out of it with a simple pymisp script + our own conversion logic
Indeed
What you said
That way we can interact with the feed’s content as if it was a misp instance
Chris Lott
@chrisInMtown_twitter
Thanks for clarifying. I will take this under advisement. It is not at all clear to me if this will let us cope with all the special situations; for example in some cases we delete the old event before ingesting a new version of it.
andras
@andras:matrix.circl.lu
[m]
Not sure. Think you folks have some pretty specific use cases that might not fit. If you see any obvious limitations holler maybe we have some ideas
Chris Lott
@chrisInMtown_twitter
well, the use cases are invariably driven by provider data oddities, the need to avoid duplicates, etc.
andras
@andras:matrix.circl.lu
[m]
Yeah duplicate management can be tough especially for sources that don’t contain unique identifiers
Chris Lott
@chrisInMtown_twitter
good day, this is probably a really dumb question, but does MISP serve out a self-documenting REST API page like what Swagger can generate with annotations? The only doc I find is from humans: https://www.circl.lu/doc/misp/automation/#search
Andras Iklody
@iglocska
that documentation is generally pretty outdated. Best to use /events/automation in MISP directly or the REST Client
OpenAPI is in the works
Chris Lott
@chrisInMtown_twitter
right, swagger is the old system, OpenAPI is the new one. Glad to hear it. I hope PHP supports annotations so the doc is always in sync with the system
1 reply
Andras Iklody
@iglocska
it doesn't per se, but @righel is working on the current state + we'll have a workflow to maintain it with each release / automatically sanity check if something is unmapped before each release
Chris Lott
@chrisInMtown_twitter
Sounds tricky. Swagger/openAPI annotations in java & python are like magic, you do a bit of work and beautiful, accurate documentation pops out. Hope your system works.
1 reply
Andras Iklody
@iglocska
it depends on the framework how supported it is less so than the language - hope so too.
andras
@andras:matrix.circl.lu
[m]
yeah I wouldn't call annotations automatic unless the annotations themselves are built automatically ^^
Chris Lott
@chrisInMtown_twitter
Oh I was not saying that the annotations are automatic! I was saying that with a modest amount of work to write the annotations in the code, the resulting documentation is really great; total effort to produce great doc is modest.
andras
@andras:matrix.circl.lu
[m]

Definitely wouldn't call that modest ;)

Also, commenting every endpoint is exactly as much effort as building an API map with all parameters, that was also @righel's point

Chris Lott
@chrisInMtown_twitter
so, my experience was with Java and Python where the only annotations I had to make were explanatory/help text for the parameters; all the endpoints, parameter names & types were available to OpenAPI from the introspection magic.
I think @luciano:matrix.circl.lu is saying that isn't available in PHP7
andras
@andras:matrix.circl.lu
[m]
exactly, what he is doing now is describing exactly that for each endpoint, just outside of the codebase.