These are chat archives for exceptionless/Discuss

6th
Nov 2015
Blake Niemyjski
@niemyjski
Nov 06 2015 00:07
Do you have skype or something
I can’t upload the file here for some reason
Blake Niemyjski
@niemyjski
Nov 06 2015 00:13
Screen Shot 2015-11-05 at 6.13.09 PM.png
@frankebersoll we may need to work on the deduplicate plugin
that’s from one guy… insane
Modesto San Juan
@modesto
Nov 06 2015 00:15
modestosja@gmail.com (email and skype accounts)
Blake Niemyjski
@niemyjski
Nov 06 2015 00:16
added
Frank Ebersoll
@frankebersoll
Nov 06 2015 05:54
@niemyjski i would like to, but i have no idea what it should exactly do. What would be considered a duplicate? When should it be filtered and when not?
Modesto San Juan
@modesto
Nov 06 2015 15:29
@niemyjski while reviewing the commit I've see this test sample: [InlineData("data.date:>now", "idx.date-d:>now", true, true)]. Can Exceptionless filter events from the future? ;)
Blake Niemyjski
@niemyjski
Nov 06 2015 15:37
@modesto yeah :) I just wanted to make sure we handle it
@frankebersoll There is an issue open on th .net repo for better stacking
but I think if we just stored a hash of the stack trace that would be good enough.
it’s what we do in the .net client
good starting point
Eric J. Smith
@ejsmith
Nov 06 2015 15:42
Morning!
Blake Niemyjski
@niemyjski
Nov 06 2015 15:43
morning birthday boy
Eric J. Smith
@ejsmith
Nov 06 2015 15:44
:-)
Guess we have to look at those changes we made last night and see if they make sense still.
Blake Niemyjski
@niemyjski
Nov 06 2015 15:44
yeah
pair?
Eric J. Smith
@ejsmith
Nov 06 2015 15:45
Cant
Frank Ebersoll
@frankebersoll
Nov 06 2015 15:53
happy birthday!
Eric J. Smith
@ejsmith
Nov 06 2015 15:54
Thanks frank.
Blake Niemyjski
@niemyjski
Nov 06 2015 15:57
@modesto looks like the skype transfer failed. I’m going to update the build to always push the artifacts to app veyor
that way you can download them from there
Frank Ebersoll
@frankebersoll
Nov 06 2015 15:57
@niemyjski storing hashes for deduplication sounds interesting, but then the error statistics don't make sense any more, right?
Blake Niemyjski
@niemyjski
Nov 06 2015 15:57
what do you mean by error statistics?
Frank Ebersoll
@frankebersoll
Nov 06 2015 15:58
well, if i log on to exceptionless and the stats tell me that one specific error happens over 9000 times, i will definitely look into it.
Blake Niemyjski
@niemyjski
Nov 06 2015 15:58
Yeah, but we only deduplicate over x seconds of time
this digest error is happening hundreds of times a second.
Frank Ebersoll
@frankebersoll
Nov 06 2015 15:59
ah!
Blake Niemyjski
@niemyjski
Nov 06 2015 15:59
and we’ve thought about keeping track of how many we’ve discarded and send that into a header to update the org stats
but that’s a different task :)
Frank Ebersoll
@frankebersoll
Nov 06 2015 15:59
that makes perfect sense then.
i thought some more about what we wanted to achieve by server side deduplication that you have implemented meanwhile but that i, shamefully, didn't test yet.
Blake Niemyjski
@niemyjski
Nov 06 2015 16:01
Did I close that issue
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:01
i found out about bloom filters recently.
Blake Niemyjski
@niemyjski
Nov 06 2015 16:01
cause I implemented that
never heard about that.
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:01
yes, we talked about it that day
Blake Niemyjski
@niemyjski
Nov 06 2015 16:01
is it like relevancy results?
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:01
it sounds very interesting - it's a data structure to find out if an element is in a set.
it needs very few bits for each element, and it is probabilistic.
meaning, it will give false positives, but you can calculate the percentage
and there are implementations for redis.
it goes like that:
you have data A and calculate hash1(A), hash2(A) and hash3(A)
result might be 1, 4 and 5
so you set bits 1, 4 and 5 to 1
01001100000
Blake Niemyjski
@niemyjski
Nov 06 2015 16:03
hmm
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:04
hash of B sets bits 4, 6 and 7 to 1
and so on
to find out if an element is in the set, you hash it and look if the bits are set
if you have a false positive, you still have to look for the item, but this will happen only 0.1% of all requests
Blake Niemyjski
@niemyjski
Nov 06 2015 16:05
that’s kinda cool
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:06
the percentage defines how many bits you need per stored item.
Blake Niemyjski
@niemyjski
Nov 06 2015 16:07
just wonder what the advantage of doing any of that server side would be
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:07
this would make it possible to check for existence of any data without hitting the database
Blake Niemyjski
@niemyjski
Nov 06 2015 16:07
already paid all of the cost except storage doing it server side
doesn’t really make too much sense
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:07
you mean the bandwidth?
Blake Niemyjski
@niemyjski
Nov 06 2015 16:08
yeah
and cpu
brb BACON!
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:12
i'm with you that we need this client side, too. added a feature branch for that.
Blake Niemyjski
@niemyjski
Nov 06 2015 16:16
back
just wondering how much we should do server side
would you want to see all the instances if they were all the same
Blake Niemyjski
@niemyjski
Nov 06 2015 16:22
guess I need to reread that issue
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:46
i don't know. we wouldn't have to transmit existing information, but update it instead
but that sounds complicated.
Blake Niemyjski
@niemyjski
Nov 06 2015 16:47
yeah
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:47
and what would be the value of it?
Blake Niemyjski
@niemyjski
Nov 06 2015 16:47
and if someone goes to look at an old instance
they wouldn’t be able to find it
Frank Ebersoll
@frankebersoll
Nov 06 2015 16:48
i wouldn't want to upload thousands of errors in short time. so we need to handle that at the client in every case.
Modesto San Juan
@modesto
Nov 06 2015 16:48
@niemyjski thanks!, the zip transfered ok and I've be able to unzip it. I've not deploy it but I'll test it tomorrow
Blake Niemyjski
@niemyjski
Nov 06 2015 16:48
well only for errors
@modesto awesome!