These are chat archives for exceptionless/Discuss

25th
Mar 2016
Eric J. Smith
@ejsmith
Mar 25 2016 01:45
So been thinking about the count issue. I don't really want to have it be a custom field. I think we are going to have to add a count property to the event. I really wanted it to just be the value field, but there are too many issues with that. We can introduce a new field and default missing values to 1.
@s
@srijken @niemyjski
Blake Niemyjski
@niemyjski
Mar 25 2016 02:02
Why not just use a extended data field
Having value and count is confusinh
Eric J. Smith
@ejsmith
Mar 25 2016 02:02
not doing an extended property for something that is going to be central to the system.
we are going to be showing it on the grids and summing it for the stacks etc.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:04
:-1:
It's only for stacks
Eric J. Smith
@ejsmith
Mar 25 2016 02:04
its not for stacks
its events
Blake Niemyjski
@niemyjski
Mar 25 2016 02:05
It's a count that represents duplicated count
It's for totals and counts on things like the stack page
Eric J. Smith
@ejsmith
Mar 25 2016 02:05
yes, and that is a central piece of the system.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:05
Doesn't make sense for an event overview page and if that's the case it doesn't make shear making it first class
Eric J. Smith
@ejsmith
Mar 25 2016 02:05
it should not be in a custom property
it does make sense for the overview page. it’s part of the event.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:06
And the label being de duplicated count?
We don't want people setting it
Eric J. Smith
@ejsmith
Mar 25 2016 02:07
yes which is why it can’t be value like I was wanting it to be.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:08
I guess my thoughts are an extended data item works just like it and is as first class as any of our other models. Moving it to the event object is going to cause index issues for all current events
Eric J. Smith
@ejsmith
Mar 25 2016 02:09
its not first class and custom field shouldn’t be used for something that will be 1st class all over the app.
no its not
missing = 1
I guess we could call it “duplicates” to make it a little clearer.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:10
Yeah
Eric J. Smith
@ejsmith
Mar 25 2016 02:12
nah, I don’t like that… it’s not really accurate.
rolling up feature usages its this event represents X feature usages.
duplicates doesn’t really make sense. Count seems like a better word.
I think count and value isn’t that confusing.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:14
I wish we could add an attribute and have Intellisense not even pick it up
Eric J. Smith
@ejsmith
Mar 25 2016 02:14
a little more generic for other use cases
Blake Niemyjski
@niemyjski
Mar 25 2016 02:15
But we don't want people ever setting it
Eric J. Smith
@ejsmith
Mar 25 2016 02:15
it will be shown and even settable. But it will be clear what it’s for.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:15
Yeah
Eric J. Smith
@ejsmith
Mar 25 2016 02:15
value is open ended to be other kinds of values
Blake Niemyjski
@niemyjski
Mar 25 2016 02:15
Guess you get the power to screw up your dashboards :)
Eric J. Smith
@ejsmith
Mar 25 2016 02:15
count will be explicitly saying that X of these things happened.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:17
What about total
Instead of count
It's the total number of events
Eric J. Smith
@ejsmith
Mar 25 2016 02:17
not as clear
Blake Niemyjski
@niemyjski
Mar 25 2016 02:17
I think it's much clearer
Total events 1
Total duplicated events 5
Eric J. Smith
@ejsmith
Mar 25 2016 02:18
how is total more clear than count. Count on event means count of event
total what?
Blake Niemyjski
@niemyjski
Mar 25 2016 02:18
Total # of events
Eric J. Smith
@ejsmith
Mar 25 2016 02:18
nah
Blake Niemyjski
@niemyjski
Mar 25 2016 02:18
So we have a counter event type
And you see count propert and a total property
Which one are you going to set ?
Just playing devils advocate cause we gotta live with tubs change forever
Eric J. Smith
@ejsmith
Mar 25 2016 02:19
count of counts
Blake Niemyjski
@niemyjski
Mar 25 2016 02:20
Lol
Eric J. Smith
@ejsmith
Mar 25 2016 02:20
that is a term used in librato
Blake Niemyjski
@niemyjski
Mar 25 2016 02:20
Ok
Eric J. Smith
@ejsmith
Mar 25 2016 02:20
total is way less clear
Blake Niemyjski
@niemyjski
Mar 25 2016 02:20
Total is the property name we use in the Ui today and the property returned by summaries
Eric J. Smith
@ejsmith
Mar 25 2016 02:20
total sounds like some sort of value got totaled up or total price for some order event or something.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:21
Yeah
I do kinda like count I just don't want someone being confused.
{type:"count", value:50, count:4}
Guess if we have no extension methods or builder methods it will be harder to find
The thing that sucks the most about this
We have todo aggregations for all of this
Instead of using response.total
Eric J. Smith
@ejsmith
Mar 25 2016 02:24
yes, we will have to use it to calculate the stack total.
we calculate stack total already
Blake Niemyjski
@niemyjski
Mar 25 2016 02:24
And totals for everything
Nope
Eric J. Smith
@ejsmith
Mar 25 2016 02:25
yes, because it’s a filtered count.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:25
We use the total retuned y elastic
We'd have to add an aggregation
Because total is number of docs matching not total + duped
Eric J. Smith
@ejsmith
Mar 25 2016 02:26
yeah, but you already made all the number stat things.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:26
Yeah I know
We'd have to ignore the total property and just add s count aggregate and pull it from numbers
Kinda excited about the change I made tonight
Going to test it tomorrow and push it
Eric J. Smith
@ejsmith
Mar 25 2016 02:27
{type:counter name:someevent count:5}
I think that makes pretty good sense.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:27
The site will be really snappy I shaved about .6 seconds from recent list and .6 seconds from dashboard stats
Eric J. Smith
@ejsmith
Mar 25 2016 02:27
cool
Blake Niemyjski
@niemyjski
Mar 25 2016 02:27
Yesh
Eric J. Smith
@ejsmith
Mar 25 2016 02:28
alright, so we are going to add a count property and move forward with this dedupe code.
Blake Niemyjski
@niemyjski
Mar 25 2016 02:28
Yeah
And I'll make the server side change
Cool thing about this is we can show in usage reports we saved you x amount of events not that someone would care :)
Count - total
Blake Niemyjski
@niemyjski
Mar 25 2016 02:55
@ejam do we want to have count be nullable or default to 1 all the time (set in ctor)
Int16?
Uint16
Eric J. Smith
@ejsmith
Mar 25 2016 03:03
nullable
int
Blake Niemyjski
@niemyjski
Mar 25 2016 03:06
Well uint is positive only
Kinda gives the intention
Blake Niemyjski
@niemyjski
Mar 25 2016 13:35
    /// <summary>
    /// The number of duplicated events.
    /// </summary>
    public uint? Count { get; set; }
could change that to int..
would we ever want a negative number in there?
guess int protects people from a person being idiot and putting a negative via json.
Eric J. Smith
@ejsmith
Mar 25 2016 13:41
I think uint has issues in some scenarios.
Uints used to get flagged for some reason by the compiler and I can't remember why.
Not CLS compliant
Uint should not be part of a public API
Blake Niemyjski
@niemyjski
Mar 25 2016 13:46
ok
good to know
int it is :D
Blake Niemyjski
@niemyjski
Mar 25 2016 14:00
@skoub I merged in latest changes for you. Was thinking I had some code in there that I could have used in my branch but comes to find out it was in a different branch hehe
Blake Niemyjski
@niemyjski
Mar 25 2016 14:22
@srijken I’m going to push these changes with count out today
but we can’t push the dedup code until 1st of the month
as any events submitted with a dedup count won’t be indexed
guess I should create an object or string property with a default value that gets injected there and can’t be set via the ui
cause we might want to set it to a date, 0, 1 or some other value
Eric J. Smith
@ejsmith
Mar 25 2016 14:49
should not be using the script if you don’t need to.
we don’t always need to default the value.
Blake Niemyjski
@niemyjski
Mar 25 2016 14:49
yeah guess but how would you know you need it
esp generically
when you some something
Eric J. Smith
@ejsmith
Mar 25 2016 14:50
where are you getting the default value?
Blake Niemyjski
@niemyjski
Mar 25 2016 14:50
not it was hardcoded to 0
otherwise sums and average values got messed up
Eric J. Smith
@ejsmith
Mar 25 2016 14:50
well that’s not ok… it will need to default to 1 for counts.
Blake Niemyjski
@niemyjski
Mar 25 2016 14:51
yeah I get that
so how should we go about setting a default script
Eric J. Smith
@ejsmith
Mar 25 2016 14:51
sum shouldn’t be messed up by missing values.
Blake Niemyjski
@niemyjski
Mar 25 2016 14:51
have hard coded logic to see if it’s a field we know
if it’s count or value field we set the script?
but you want sum default value on the count field
Eric J. Smith
@ejsmith
Mar 25 2016 14:52
make missing value part of FieldAggregation
Blake Niemyjski
@niemyjski
Mar 25 2016 14:52
to be a nullable int?
Eric J. Smith
@ejsmith
Mar 25 2016 14:52
yes, when summing the count we need to default it to a value… but when summing the value field we should not default.
Blake Niemyjski
@niemyjski
Mar 25 2016 14:52
and how does it get set
yeah I know
i get that
so should I special case it inside of the dumb model?
or in the event stats code
Eric J. Smith
@ejsmith
Mar 25 2016 14:54
Add a nullable missing value to the FieldAggregation and then if it’s populated use the default value script.
what do our expressions look like again?
Blake Niemyjski
@niemyjski
Mar 25 2016 14:55
avg:field
or
term:field:-exclude or term:field:include
guess we could add a third value to do
avg:field:default
kinda sucks
but terms wouldn’t need it
Eric J. Smith
@ejsmith
Mar 25 2016 15:03
what is the exclude and include?
Blake Niemyjski
@niemyjski
Mar 25 2016 15:03
we use it to filter out fixed on the new query
wait
Eric J. Smith
@ejsmith
Mar 25 2016 15:04
wouldn’t that be part of the filter itself?
Blake Niemyjski
@niemyjski
Mar 25 2016 15:04
^ (Field?.GetHashCode() ?? 0)
'distinct:stack_id,term:is_first_occurrence:-F'
sorry first occurrence
we do and it’s apart of teh filter
  • says exclude
-
otherwise it’s parsed as an include
so guess we add an extra : eh
for default value
Eric J. Smith
@ejsmith
Mar 25 2016 15:09
I’m confused
Blake Niemyjski
@niemyjski
Mar 25 2016 15:09
about?
Eric J. Smith
@ejsmith
Mar 25 2016 15:09
about your syntax
Blake Niemyjski
@niemyjski
Mar 25 2016 15:09
what’s so confusing about it
operation:field
Eric J. Smith
@ejsmith
Mar 25 2016 15:10
distinct:stack_id is great
Blake Niemyjski
@niemyjski
Mar 25 2016 15:10
operation:field:(include or -exclude)
Eric J. Smith
@ejsmith
Mar 25 2016 15:10
what is the heck is the include or exclude?
Blake Niemyjski
@niemyjski
Mar 25 2016 15:11
case FieldAggregationType.Term:
var termField = field as TermFieldAggregation;
if (termField == null)
throw new InvalidOperationException("term aggregation must be of type TermFieldAggregation");
                    aggregation.Terms(field.Key, t => {
                        var tad = t.Field(field.Field);
                        if (!String.IsNullOrEmpty(termField.ExcludePattern))
                            tad.Exclude(termField.ExcludePattern);

                        if (!String.IsNullOrEmpty(termField.IncludePattern))
                            tad.Include(termField.IncludePattern);

                        return tad;
                    });
                    break;
term:is_first_occurrence:-F
-F is an exclude here
only happens on term queries
term aggregations*
so I guess we add support for a operation:field:default value where operation != term
that way you can specify a default date (when doing min or max) or you can specify a default value for your field if it’s missing
and if we can’t parse a date or int do we allow it
I wonder if we should be scrubing all these fields...
I guess we have an allow only list...
but not for exlcude and includes
or defaults
Sander Rijken
@srijken
Mar 25 2016 15:19
@niemyjski that makes sense (1st of the month)
we should also look at the .javascript deduper
Blake Niemyjski
@niemyjski
Mar 25 2016 15:20
Yeah
Eric J. Smith
@ejsmith
Mar 25 2016 15:24
ok I see what you are doing now… but not sure about it.
I guess you are trying to get back to a specific number, right?
Sander Rijken
@srijken
Mar 25 2016 15:31
in .javascript is a feature/di branch, is that syntax some sort of convention for something?
Blake Niemyjski
@niemyjski
Mar 25 2016 15:32
Yeah
Sander Rijken
@srijken
Mar 25 2016 15:32
any idea what? :P
Blake Niemyjski
@niemyjski
Mar 25 2016 15:33
We label all our branches with feature/ for feature branches
Sander Rijken
@srijken
Mar 25 2016 15:33
oh it’s something you do
Blake Niemyjski
@niemyjski
Mar 25 2016 15:33
@frank was playing around in there
Sander Rijken
@srijken
Mar 25 2016 15:33
I’ve watched a course on gitflow the other day
seems powerful
Blake Niemyjski
@niemyjski
Mar 25 2016 15:33
@ejsmith so do we want to allow people to specify defaults and validate them somehow
It's what we try and use :)
Sander Rijken
@srijken
Mar 25 2016 15:34
I’m gonna set that up for one of our projects @ work
most stuff is still in svn
Blake Niemyjski
@niemyjski
Mar 25 2016 15:38
You can import it all
Sander Rijken
@srijken
Mar 25 2016 15:41
I know
is it possible to merge 2 git repositories? in case it’s ever needed
Blake Niemyjski
@niemyjski
Mar 25 2016 15:44
Idk
Maybe?
Email github support
it’s simple
Blake Niemyjski
@niemyjski
Mar 25 2016 16:34
@ejsmith guess I’m adding a third part to the queries that I’ll validate to be an integer
Blake Niemyjski
@niemyjski
Mar 25 2016 18:28
just deleted 60+ lines of stats code :D
feels so good
buy what you need
instead of buy everything and use nothing
will be doing a pull request on this.. updating unit tests now.. want an extra pair of eyes on this..
Eric J. Smith
@ejsmith
Mar 25 2016 18:30
Ok
Sounds good. Love deleting code! :-)
Eric J. Smith
@ejsmith
Mar 25 2016 19:18
setting up a windows server… wow… forgot how back windows server 2012 is.
how bad
dig through the 4 levels deep set of features to try and get IIS / ASP.NET / Websockets installed, use the sweet metro full screen start menu, install the 1000 updates, try to use the browser to download something and get hit with 1000 security prompts...
Blake Niemyjski
@niemyjski
Mar 25 2016 19:23
yeah
Eric J. Smith
@ejsmith
Mar 25 2016 19:23
I saw some rumors that Windows is going to blow us away
I am hoping that was about Windows Server 2016 and it’s going to stop sucking.
so nano server will only run .net core though… so not really sure why I would use that over linux which has infinitely better management tools and hosting options.
Blake Niemyjski
@niemyjski
Mar 25 2016 19:25
yeah
who knows
only a few more days to wait
I hope it’s good too
Eric J. Smith
@ejsmith
Mar 25 2016 19:26
yeah, I think they will have a new release for build, but they are only on preview 4.
so I feel like it’s still going to be a while before RTM
Blake Niemyjski
@niemyjski
Mar 25 2016 19:27
yeah
I think rc2 isn’t going to hit it
doesn’t feel like they have any tooling out
and they should of at least had a preview or beta
..
Eric J. Smith
@ejsmith
Mar 25 2016 19:27
yeah, rc2 isn’t going to make it.
Blake Niemyjski
@niemyjski
Mar 25 2016 19:27
I don’t think you can hit an rc mark without tooling
Eric J. Smith
@ejsmith
Mar 25 2016 19:28
from the standups it sounds like they are still in chaos after changing the cli
no way it will make it for next week.
even if everything is working… no way they can call it rc again.
Blake Niemyjski
@niemyjski
Mar 25 2016 19:33
yeah
stupid
is what it is
like what they are doing but why would they even give it an rc title
they made so much more work and headaches for themselves when they did that
Eric J. Smith
@ejsmith
Mar 25 2016 19:33
I feel like it’s gonna be atleast a year before .net core is really viable.
Blake Niemyjski
@niemyjski
Mar 25 2016 19:33
now people are going to be pissed because they targeted an rc and everything changed
so they will have a lot of work todo to move to rc2
Eric J. Smith
@ejsmith
Mar 25 2016 19:34
yeah
Blake Niemyjski
@niemyjski
Mar 25 2016 19:37
did you hear about npm breakage and what caused it
kinda hillarious
now there is a group of people building a new npm cause they don’t agree with how npm handled the whole thing
tldr: guy had like 217 packages and one of them was called kik.. Well this corporation wanted it for there own project. and he said no and they threatened lawsuit and so he was like I don’t care.. well npm just transfered ownership.. so the guy delisted all of his packages breaking a crap ton of people cause bable and others depeneded on it..
Blake Niemyjski
@niemyjski
Mar 25 2016 19:58
thank god for unit tests :)
Blake Niemyjski
@niemyjski
Mar 25 2016 20:19
@ejsmith just realized something
our update all method never invalidates the cache
Blake Niemyjski
@niemyjski
Mar 25 2016 20:31
fixed that
new issue
add doesn’t invalidate the cache :(
I’m caching a value against org id
so when I add a new project I want to invalidate that
kinda sucks
what are your thoughts
guess I could wire up to adding event and clear it out myself
kind of an advanced use case
Eric J. Smith
@ejsmith
Mar 25 2016 20:36
yeah, the npm thing was so 3 days ago. :-)
Eric J. Smith
@ejsmith
Mar 25 2016 21:03
so here is something positive for MS… a lot of things in Azure are a LOT nicer than AWS.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:04
yeah
not liking it
?
I know one major thing aws has going for it
Eric J. Smith
@ejsmith
Mar 25 2016 21:05
they both have their pros and cons… but AWS is a LOT more complicated and less polished.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:06
they don’t have an * * ** who works on there dashboards.. I’ve lost my dashboards 7 SEVEN siete septem epta TIMES
Eric J. Smith
@ejsmith
Mar 25 2016 21:06
never lost mine once… pretty weird.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:06
bills lost his a few times
lately it’s been small things like 3 out of 20 charts are missing and then everythings shifted
and I complain and it’s restored
not sure if it’s a troll engineer at this point
Eric J. Smith
@ejsmith
Mar 25 2016 21:07
haha… yeah, you probably pissed them off and they are just messing with you.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:07
yeah
lets activate his web cam and watch his blood pressure
haha
Blake Niemyjski
@niemyjski
Mar 25 2016 21:13
@ejsmith can you look at this
exceptionless/Exceptionless#205
need to do a quick ui test around it but I’m 99% sure all is good
looked at both ui code and server side usages when I was working on it
Blake Niemyjski
@niemyjski
Mar 25 2016 21:21
OMG
the github diff unified diff option is a godsend
<3 it
Blake Niemyjski
@niemyjski
Mar 25 2016 21:27
unreal how much faster this is locally
dang
some perspective
feels like I’m filtered to last hour view
Eric J. Smith
@ejsmith
Mar 25 2016 21:28
12X faster?
Blake Niemyjski
@niemyjski
Mar 25 2016 21:28
when I’m filtered all time
Eric J. Smith
@ejsmith
Mar 25 2016 21:28
what are you specifically wanting me to look at?
Blake Niemyjski
@niemyjski
Mar 25 2016 21:28
well the stats query and most frequent query were both shaved by .6 seconds against prod cluster when filtered to the exceptionless project last 30 days
if that says anything
quick look at the stats changes / daily summary job
see if anythign looks off
Eric J. Smith
@ejsmith
Mar 25 2016 21:29
well, there are 55 changed files
that is not a quick look
Blake Niemyjski
@niemyjski
Mar 25 2016 21:30
I know I had to fix 2 bugs in foundatio repositories.. :( wanted a small commit log but then had to update foundatio
added unit tests for all my changes
so I think we are safe
found a bug where document id could be null on insert
Blake Niemyjski
@niemyjski
Mar 25 2016 21:38
@ejsmith you can filter by commit
that would have saved a ton of time
lol
I’m mergingi t
kinda excited
these changes will allow me to add % of affected users to the stack lists in no time flat
:D
Eric J. Smith
@ejsmith
Mar 25 2016 21:42
that would be pretty cool.
we have to figure out how to do it at the list level.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:43
yeah
Eric J. Smith
@ejsmith
Mar 25 2016 21:43
need to be able to see that when viewing the list of stacks and sort by the stacks affecting the most users.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:43
very easily now
yeah
Eric J. Smith
@ejsmith
Mar 25 2016 21:43
yeah?
done in no time, eh?
Blake Niemyjski
@niemyjski
Mar 25 2016 21:44
you ready for this
Eric J. Smith
@ejsmith
Mar 25 2016 21:44
?
Blake Niemyjski
@niemyjski
Mar 25 2016 21:44
Screen Shot 2016-03-25 at 4.39.54 PM.png
before
all time exceptions for our org
Screen Shot 2016-03-25 at 4.43.48 PM.png
after :)
same view
Eric J. Smith
@ejsmith
Mar 25 2016 21:45
nice
Blake Niemyjski
@niemyjski
Mar 25 2016 21:45
can’t wait to look at the elastic dashboards monday
I’d guess we probably saved us some cpu usage
hehe
Eric J. Smith
@ejsmith
Mar 25 2016 21:45
nice
weird how the session events list takes a little while to populate.
seems like it would be a very simple query.
Blake Niemyjski
@niemyjski
Mar 25 2016 21:47
yeah