Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 09 08:38
    adulau commented #267
  • May 09 08:38
    adulau closed #266
  • May 09 08:38

    adulau on main

    chg: [warninglists] Restructure… Merge pull request #267 from Wa… (compare)

  • May 09 08:38
    adulau closed #267
  • May 08 18:29
    Wachizungu opened #267
  • May 05 09:13
    Wachizungu edited #245
  • May 05 06:05
    adulau labeled #266
  • May 04 22:23
    Wachizungu opened #266
  • May 04 16:25
    adulau commented #265
  • May 04 16:25

    adulau on main

    chg: [User stories] Fix typo Merge pull request #265 from Wa… (compare)

  • May 04 16:25
    adulau closed #265
  • May 04 09:24
    Wachizungu opened #265
  • May 01 21:01
    Wachizungu commented #264
  • May 01 20:42
    adulau commented #264
  • May 01 20:42

    adulau on main

    chg: [FAQ] add sudo with user w… Merge pull request #264 from Wa… (compare)

  • May 01 20:42
    adulau closed #264
  • May 01 18:41
    Wachizungu opened #264
  • Apr 24 13:07

    adulau on main

    chg: [automation] add doc for /… Merge pull request #263 from Wa… (compare)

  • Apr 24 13:07
    adulau closed #263
  • Apr 24 12:10
    Wachizungu opened #263
Chris Lott
@chrisInMtown_twitter
we would like overlap (correlation) analysis for object attributes also. Today we only get that for plain attributes
andras
@andras:matrix.circl.lu
[m]
Oh wow really? That’s a bug
It should be for everything
You mean via feed/server caching?
Chris Lott
@chrisInMtown_twitter
um let me rephrase please. I'm not saying it cannot. I'm checking if it CAN
andras
@andras:matrix.circl.lu
[m]
Generally it should ignore objects altogether and fetch all attribute values
Meaning it flattens the event first
Object attributes and normal ones alike
Chris Lott
@chrisInMtown_twitter
none of our feeds processed by supplied MISP/Cake/delta ingest use object-attributes. Only our custom feeds; and MISP/Cake cannot fetch/cache those
andras
@andras:matrix.circl.lu
[m]
It should be able to cache those too
Unless i completely missed the point
Chris Lott
@chrisInMtown_twitter
that's why I say "MISP/Cake cannot cache"
The problem is the ETL task of consuming the premium data feed and munging it into events and attributes;
andras
@andras:matrix.circl.lu
[m]
Yeah depending on the format that might need a fair bit of glue
Chris Lott
@chrisInMtown_twitter
hmm questioning my assumptions here .. I believe that "cache" is an action that copies from the source (the remote server) to the local redis cache.
andras
@andras:matrix.circl.lu
[m]
It’s even dumber than that
Chris Lott
@chrisInMtown_twitter
If however "cache" can operate as copying from local MISP database of events, obj & attributes to local Redis memcache, then that would be very interesting
andras
@andras:matrix.circl.lu
[m]
It loops the remote through the ingestion that you would normally use for a fetch operation
Extracts the values from the derived attributes
Hashes them
And throws them into Redis
When it comes to misp format feeds/misp servers
It does something different
Chris Lott
@chrisInMtown_twitter
Thanks for the details. but for our in-house-developed ingestion scripts that copy remote to local database, I don't know how to get that data into the redis cache/analysis chain
andras
@andras:matrix.circl.lu
[m]
What format are they in?
Chris Lott
@chrisInMtown_twitter
some of our in-house ingestion scripts create object attributes
oh boy, format. Some XML, some JSON
andras
@andras:matrix.circl.lu
[m]
That should be fine
If you convert them to misp format already
You’re half way there
Generate a misp feed out of them
Chris Lott
@chrisInMtown_twitter
hmm starting to question everything we're doing :/
andras
@andras:matrix.circl.lu
[m]
Hehe
Chris Lott
@chrisInMtown_twitter
we use Python & PyMISP to massage the data and inject straight to events & attributes & objects
andras
@andras:matrix.circl.lu
[m]
So basically what we do normally
If we have a source in a format misp can’t cope with
Chris Lott
@chrisInMtown_twitter
I think you're suggesting, if we consume a premium feed and transform it to a MISP-friendly format, then we could use the MISP ingester, which would also allow us to use the MISP cacher. Did I understand correctly?
andras
@andras:matrix.circl.lu
[m]
We generate a misp feed out of it with a simple pymisp script + our own conversion logic
Indeed
What you said
That way we can interact with the feed’s content as if it was a misp instance
Chris Lott
@chrisInMtown_twitter
Thanks for clarifying. I will take this under advisement. It is not at all clear to me if this will let us cope with all the special situations; for example in some cases we delete the old event before ingesting a new version of it.
andras
@andras:matrix.circl.lu
[m]
Not sure. Think you folks have some pretty specific use cases that might not fit. If you see any obvious limitations holler maybe we have some ideas
Chris Lott
@chrisInMtown_twitter
well, the use cases are invariably driven by provider data oddities, the need to avoid duplicates, etc.
andras
@andras:matrix.circl.lu
[m]
Yeah duplicate management can be tough especially for sources that don’t contain unique identifiers
Chris Lott
@chrisInMtown_twitter
good day, this is probably a really dumb question, but does MISP serve out a self-documenting REST API page like what Swagger can generate with annotations? The only doc I find is from humans: https://www.circl.lu/doc/misp/automation/#search
Andras Iklody
@iglocska
that documentation is generally pretty outdated. Best to use /events/automation in MISP directly or the REST Client
OpenAPI is in the works
Chris Lott
@chrisInMtown_twitter
right, swagger is the old system, OpenAPI is the new one. Glad to hear it. I hope PHP supports annotations so the doc is always in sync with the system
1 reply
Andras Iklody
@iglocska
it doesn't per se, but @righel is working on the current state + we'll have a workflow to maintain it with each release / automatically sanity check if something is unmapped before each release