FYI: The CMS serde I came up with is captured in https://github.com/confluentinc/examples/blob/3.2.x-algebird/kafka-streams/src/main/scala/io/confluent/examples/streams/algebird/TopCMSSerde.scala. It's just a few lines of code, essentially it's two calls to Chill's ScalaKryoInstantiator. If you have a minute to spare, feel free to let me know if that is how Chill should be used here.
for context, we have several open PRs that breaks it. typelevel/cats#1712 typelevel/cats#1527 typelevel/cats#1878
@kailuowang
I'll look at them.
Hi,
I am new to these algorithms and curious about the differences between Sliding HyperLogLog(https://hal.archives-ouvertes.fr/hal-00465313/file/sliding_HyperLogLog.pdf) vs HyperLogLog Series. I want to create thousands of counters for providing sliding window based cardinality estimation. e.g. cardinality estimate for last 30 days, last 7 days and last 24 hours.
- Will the hyperloglog series evict/forget the older data for example in my case data older then 30 days?
- Does the size of hyperloglog series increase with time? For a 12 bit counter, what would be the minimum and maximum size?
I am trying to find the answers to these questions and it would be really helpful if I get a quick response.
Thanks.
Hey, I have a case class Thing(name: String). I need to "reduce" a Set[Thing] into a Set[Thing] where resultant set is the one with a max count of identical names. That is,
Set(Thing("Cory"), Thing("Cory"), Thing("Ahmad"), Thing("Kevin"), Thing("Kevin")) "reduces" toSet(Thing("Cory"), Thing("Cory"), Thing("Kevin"), Thing("Kevin")).
How do I neatly put this into one of the structures defined in algebird?
An empty set won't reduce to anything in my case, so there is no identity element here. But, it is also not evident to me how it falls into a semigroup since the result of the reduction is "many" not "one".
Max looked promising at first but I still don't see how to leverage it.
I'm not 100% sure what you want is actually associative... which means it may not be a semigroup/monoid
you can write a Fold, which is more general, but sorry I don't immediately see an answer to your problem
i think representing as
Map[K, Long] where you keep track of the counts, might make it clearer
but pruning that is not associative.
we have cms with topk, but it is only approximately associative
it is related to a count min sketch
@xennygrimmato we do have beginner tags: https://github.com/twitter/algebird/issues?q=is%3Aissue+is%3Aopen+label%3Abeginner
@xennygrimmato I think the link was to a different line when I made it, and now commits have changed what is on that line. I think I was pointing here: https://github.com/twitter/algebird/blob/develop/algebird-test/src/test/scala/com/twitter/algebird/HyperLogLogTest.scala#L57 basically, all the random HLL's we create have a single element in them. That is not great. Really, we should generate a list of longs, for instance, and add all of them to the HLL. That would be a better test. I expect the tests to still pass, but it is something we should think about: high quality and good coverage random generation.
@matfed there is this: https://github.com/twitter/algebird/blob/develop/algebird-core/src/main/scala/com/twitter/algebird/MapAlgebra.scala#L196 we could do with a syntax enrichment package probably. That would be nice to add.
@johnynek Is there some documentation for the SparseHLL case class? I want to understand what
Specifically, I want to know the use of
maxRhow represents here: https://github.com/twitter/algebird/blob/develop/algebird-core/src/main/scala/com/twitter/algebird/HyperLogLog.scala#L393Specifically, I want to know the use of
Max[Byte] there.
I was wondering how can you serialize a Bloom Filter instance
you know to write it out somewhere
then retrieve it and deserialize as BF[String] for example
Kryo would probably work fine for it... the basic idea is just serialize the indices of all the bits.
(that are set)
well... actually that's not true... only the value type.
then you just iterate through the table and serialize each of the Vs
if the received knows the table dimensions, you just do that loop on the receiving side
if you don't know the size (which I would recommend you do statically know in your app) you can write the dimensions of the table first, then all the V values
Hi all, I'd love to contribute to this library. This issue in particular seems like it would be fun to tackle: twitter/algebird#562 . It was posted a couple years ago, so I was wondering if people still want this change to happen. Just wanted to check in before I spent a bunch of time on it.
Hi folks! I'd really like to treat an
ExpHist as a Monoid, and also serialize it in and out. I've essentially solved the serialization issue, but there doesn't seem to be a Monoid instance for ExpHist, even though it has empty and add. Is there some issue around the laws I should be aware of? I see that I can take approximateSum, which gives me an Approximate, so that might work, if there were an easy way to go back from Approximate to ExpHist for the sake of the serialization. Any and all thoughts are appreciated. :-)