These are chat archives for exceptionless/Discuss

24th
Mar 2016
Blake Niemyjski
@niemyjski
Mar 24 2016 00:13
HOLY CRAP
our deployment size wentfrom 113mb to 13.8
had to double check the artifacts to ensure we had everything
lol
Blake Niemyjski
@niemyjski
Mar 24 2016 00:37
@/all new release is out for both the ui and server (contains the ui).
I still need to tweak the change log a bit...
but there are a ton of good fixes and improvements there
Adam Zolotarev
@adamzolotarev
Mar 24 2016 02:06
Thanks for the new release!
I was trying it out and I’m a little confused about the new search bar. What happened to the Include Hidden and Include Fixed checkboxes? Is the only way to show hidden/fixed now only by using hidden:true and fixed:true? Is there an easy way to show all events no matter their status?
Eric J. Smith
@ejsmith
Mar 24 2016 02:19
Yeah, @adamzolotarev currently have to do that in the search bar to see all. We were trying to figure out if we needed to do something special to allow viewing all events. Do you have any ideas?
Blake Niemyjski
@niemyjski
Mar 24 2016 02:23
(hidden:fixed OR hidden:true OR fixed:true OR fixed:false)
Is what you'd need to run.. Could just have the Ui check for hidden:any and remove it..
Adam Zolotarev
@adamzolotarev
Mar 24 2016 02:36
Ok, thanks for a quick response!
Blake Niemyjski
@niemyjski
Mar 24 2016 02:37
No problem. We were hoping to simplify everything and make filtering more obvious
The downside is it made this one scenario a bit harder but we felt like not that many people would normally view all events
Eric J. Smith
@ejsmith
Mar 24 2016 02:41
I think we need some way to easily view all eve
Adam Zolotarev
@adamzolotarev
Mar 24 2016 02:42
yeah, makes it a little more difficult for the way we use it. When we start working on an issue we mark it as Hidden in Exceptionless. That way people have an easy way to filter out issues which are being addressed. But we still want to see it for monitoring purposes. But I don't know how many people use it the way we do.
Eric J. Smith
@ejsmith
Mar 24 2016 02:42
It's not intuitive that it's filtered
Adam Zolotarev
@adamzolotarev
Mar 24 2016 02:43
Either way, thanks for the new release and new features!
Eric J. Smith
@ejsmith
Mar 24 2016 02:43
Yeah, need to think about a way to make it easier and more intuitive.
We mostly wanted to make it more obvious what you are filtered to. Because before the date range was hidden and if you had a filter it was hidden too.
Adam Zolotarev
@adamzolotarev
Mar 24 2016 02:44
yeah, I really like that you can see the date range now
Eric J. Smith
@ejsmith
Mar 24 2016 02:44
Problem is that now it's not intuitive that you are filtered by default. :-(
Blake Niemyjski
@niemyjski
Mar 24 2016 02:49
Well
We could put the hidden and fixed query in there by default
And if you remove it it's gone until your next session
Kinda stinks
Eric J. Smith
@ejsmith
Mar 24 2016 02:53
Yeah, I'm not sure what to do. Another option is to add a drop down to the right of the filter box with the hidden and fixed options on it.
Another option is to have saved filters.
Where you click on the filter box and a drop down shows letting you pick a saved filter.
Blake Niemyjski
@niemyjski
Mar 24 2016 02:57
Yeah :(
If it's populated by default to exclude things people might be more curious and inclined to mess with it
Eric J. Smith
@ejsmith
Mar 24 2016 03:02
Maybe but I don't really like it. Gotta think about it.
Sander Rijken
@srijken
Mar 24 2016 07:24
Why not put 2 toggle buttons in the navbar?
Blake Niemyjski
@niemyjski
Mar 24 2016 11:10
We wanted to keep it clean
If you think about it if you hide something and want to see it then you just put hidden:true
For a large percentage of users never click all
Although that's an assumption
Sander Rijken
@srijken
Mar 24 2016 14:10
maybe a dropdown with the last few searches, followed by a divider, followed by the labels of some predefined searches
could also be used to add more things, and maybe add saved searches
could even imagine having my own saved searches, and saved searches that I can share with the rest of the organization
Adam Zolotarev
@adamzolotarev
Mar 24 2016 14:20
maybe add a user-level setting to either show or hide the two checkboxes? Then only people who care about it would enable it. Of course, this would increase complexity.
Blake Niemyjski
@niemyjski
Mar 24 2016 14:21
well
if you want to see hidden
then just type hidden:true
but I agree it would be nice to have saved searched
searches*
Adam Zolotarev
@adamzolotarev
Mar 24 2016 14:24
sometimes I want to look for something across all stacks, no matter hidden, fixed or not. This makes the query somewhat cumbersome: "(hidden:true OR hidden:false OR fixed:true OR fixed:false) AND myOtherQuery"
Blake Niemyjski
@niemyjski
Mar 24 2016 14:24
yeah'
the root issue is the ui adds hidden:false if you don’t have hidden in your query
so if we just removed that code and defaulted to hidden:false and fixed:false and allowed you to remove it you’d see everything
Eric J. Smith
@ejsmith
Mar 24 2016 15:02
@adamzolotarev agreed. It’s too cumberson right now. I think what needs to happen is that we need to get rid of the filtered views in the left nav and switch to saved filters.
because it’s also confusing that you click on things in the left nav and they are filtering your data as well.
Eric J. Smith
@ejsmith
Mar 24 2016 15:07
I think if we had saved filters then we could just create pre-defined saved filters for you for “Exceptions”, “Broken Links”, “Feature Usages” etc
and since our event types are open ended, then people can create their own saved views or delete ones they don’t care about.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:12
yeah
could simplify the ui
but it’d have to be done really well
Eric J. Smith
@ejsmith
Mar 24 2016 15:12
yep
Adam Zolotarev
@adamzolotarev
Mar 24 2016 15:14
so, if I care about all events, I could have something like "All Kind of Events" view. That sounds interesting. Would you still be able to search within a given view?
Eric J. Smith
@ejsmith
Mar 24 2016 15:14
I think we have a drop down button on the left of the filter box and also if you click into the filter box a panel flys out below with a list of saved filters and a button to save a filter.
so the event type views on the left nav would go away and you would just have the All Events views
then you combine that with saved filters for “Exceptions"
Adam Zolotarev
@adamzolotarev
Mar 24 2016 15:21
oh I see. Yeah, I think it would be nice. Pretty big change for UI
Blake Niemyjski
@niemyjski
Mar 24 2016 15:23
massive :)
Eric J. Smith
@ejsmith
Mar 24 2016 15:34
Well would be a lot of removed things and simplified, but coding would just be adding the filter drop down and saving filters.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:36
yeah
and update all routing to set the filter and redirect the view
Eric J. Smith
@ejsmith
Mar 24 2016 15:37
That would be removing code, right?
Blake Niemyjski
@niemyjski
Mar 24 2016 15:37
yes and no
long term yes
still have to figure out how we’d want to navigate to to different dashboards
for example
if you are on manage organization page and you want to view that orgs dashboard
how do you navigate to it
do we call the existing route which sets the org and redirects keeping your new filter
or do we pass the org filter on a query string and redirect?
we’d want to keep the old routes and we’d have to update all the email templates
and phase them out over time :)
Eric J. Smith
@ejsmith
Mar 24 2016 15:39
Yeah
Blake Niemyjski
@niemyjski
Mar 24 2016 15:39
would make routing much easier just have decisions to make
Eric J. Smith
@ejsmith
Mar 24 2016 15:40
Could change it to pick a saved view by route.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:40
and then what do we do with the typed routes etc
and have predefined system routes that can’t be deleted?
Eric J. Smith
@ejsmith
Mar 24 2016 15:40
Could actually put the saved filter name in the route and have urls that are similar.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:41
but then there goes linkability if we redirect to a simple route
eh, might be hard
what if you call a saved filter sessions
and now you have two /sessions/dashboard
and how do you share that if someone doesn’t have a filter with that name
almost be better if we just redirected with the full faw search query as a query string in the url
Eric J. Smith
@ejsmith
Mar 24 2016 15:43
they are stored with the project.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:43
and parsed it and applied that to dropdowns and the search bar
Eric J. Smith
@ejsmith
Mar 24 2016 15:43
yeah, I think that is the backup if the filter isn’t saved.
but I don’t want fugly urls if we can use saved filter names.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:44
so you’d save a filter on a project? and what an org too?
Eric J. Smith
@ejsmith
Mar 24 2016 15:45
ahh true… they are across orgs.
maybe the easy 1st step is just encode the filter in the url and deal with ugly urls
/dashboard?f=type:error
that isn’t too horrible
Blake Niemyjski
@niemyjski
Mar 24 2016 15:47
no
Eric J. Smith
@ejsmith
Mar 24 2016 15:47
no?
Blake Niemyjski
@niemyjski
Mar 24 2016 15:47
and store the filters in local storage
no that’s not too terrible :)
Eric J. Smith
@ejsmith
Mar 24 2016 15:48
yeah, so not even store them server side, eh?
I think that has the potential to really piss people off when they create filters and lose them.
but it is a simple 1st step
Blake Niemyjski
@niemyjski
Mar 24 2016 15:48
kinda sucks for going to a different machine but you could save the url
yeah
Eric J. Smith
@ejsmith
Mar 24 2016 15:48
yeah, it does suck
Blake Niemyjski
@niemyjski
Mar 24 2016 15:49
could just store them on the user as extended data
Eric J. Smith
@ejsmith
Mar 24 2016 15:49
even storing them with the user sucks
because I want to setup a filter and have other people see it.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:49
yeah
Eric J. Smith
@ejsmith
Mar 24 2016 15:49
this whole cross org shit is killing us.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:49
yeah
I know
complicates a lot of things
Eric J. Smith
@ejsmith
Mar 24 2016 15:52
yeah it really does.
so you save to org and then it’s which org to save to and what if 2 orgs have filters with the same name?
Blake Niemyjski
@niemyjski
Mar 24 2016 15:53
yeah
well I think you could show a dropdown with different org filters.
with groups like how our project filter is
Eric J. Smith
@ejsmith
Mar 24 2016 15:54
too complicated.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:54
and then when you save it you are asked which one to save it to
Eric J. Smith
@ejsmith
Mar 24 2016 15:54
I want to get simpler, not more complicated.
Blake Niemyjski
@niemyjski
Mar 24 2016 15:54
would be interesting to know how many people are in multiple organizations
Eric J. Smith
@ejsmith
Mar 24 2016 15:55
yeah
Blake Niemyjski
@niemyjski
Mar 24 2016 15:55
let me see if I can figure that out
:)
Eric J. Smith
@ejsmith
Mar 24 2016 15:55
and then how many of those are because they want to restrict peoples access.
if we had security and billing at the project level then it would probably work for 99%
and a lot of people before wanted to view data across projects.
but all of that stuff is a bunch of work… so we gotta figure out what the baby steps are.
Blake Niemyjski
@niemyjski
Mar 24 2016 16:00
yeah
I completely agree
Eric J. Smith
@ejsmith
Mar 24 2016 16:02
I say we just save the filter in the url and you bookmark it.
But we need to pre-populate with common filters.
Blake Niemyjski
@niemyjski
Mar 24 2016 16:04
how does this solve the initial problem tho
fixed:true hidden:false
is that going to be included in every query going forward
Eric J. Smith
@ejsmith
Mar 24 2016 16:06
Thinking that we have to have named filters. Because I want to have an All Events that you can just pick.
Blake Niemyjski
@niemyjski
Mar 24 2016 16:07
that could be an option that just sets teh filter to that
Eric J. Smith
@ejsmith
Mar 24 2016 16:08
so what if we save filters to users docs but still encode the filter in the URL so that you can send it to people.
Eric J. Smith
@ejsmith
Mar 24 2016 16:18
only thing that sucks is that people have to paste urls to share saved filters.
but can live with that.
I think we need to add some filter helpers too… like maybe support “*” for all events.
and adding the fixed:any and hidden:any
we can just do a string replace to transform those… fixed:any -> exists:fixed
exists :fixed
damn markdown
Blake Niemyjski
@niemyjski
Mar 24 2016 17:21
yeah
could do that
what if you wanted to do
user:blake*
would the * be detected client side or would we still send fixed:false hidden:false through
or I guess you’d have to do * with whitespace on either side
Eric J. Smith
@ejsmith
Mar 24 2016 17:56
Has to be * by itself
Adam Zolotarev
@adamzolotarev
Mar 24 2016 18:00
This message was deleted
Blake Niemyjski
@niemyjski
Mar 24 2016 18:05
@frankebersoll I was just running the sample to check something out and got this.. have you seen this:
Screen Shot 2016-03-24 at 1.04.46 PM.png
Blake Niemyjski
@niemyjski
Mar 24 2016 18:11
So I’m looking at one of our competitors… and I can’t believe with all the resources they have hardly anything is real time..
also, they don’t show you how many users are affected by an error
and I have an excellent health score and 100% happy users even though I have one user (me) whose had 4 errors during a single session
I’d say my satisfaction would be piss poor
Blake Niemyjski
@niemyjski
Mar 24 2016 18:19
@BillHenning if you were looking at a stack
like list
Screen Shot 2016-03-24 at 1.20.22 PM.png
should the most frequent be most frequently occuring events
or events that affect the most users
or both
also
would you want to see a % of users affected
like 83% with a tooltip that shows the number of users affected with maybe a link to view those users?
idk keeping it simple, not sure if we want to display a list of users at this point
would the top occuring exception be more important than one that effects the most amount of users
what if the top occuring exception indirectly effects the most users?
Sander Rijken
@srijken
Mar 24 2016 19:36
I think we should also discuss what needs to happen on the server end before we can enable deduplication in .Net (if any)
the other choice that’s still open is the pros and cons of using Value vs a special field for dedupe
Eric J. Smith
@ejsmith
Mar 24 2016 19:39
yeah, need to make a decision on that so we can wrap that project up. :-)
Sander Rijken
@srijken
Mar 24 2016 19:40
I ran into a crazy storm of events this morning
Eric J. Smith
@ejsmith
Mar 24 2016 19:40
I really like the simplicity of using value, but there are definite things that might cause problems with it.
yeah?
Sander Rijken
@srijken
Mar 24 2016 19:40
couple of thousand in an hour
Eric J. Smith
@ejsmith
Mar 24 2016 19:41
dedupe would’ve worked good, eh?
Sander Rijken
@srijken
Mar 24 2016 19:42
i’m sure
it will change how things look in the UI though
when browsing the events in the stack, you won’t see 5 pages of “a few seconds ago"
but instead just one.. but you might want to see the value there as well
Eric J. Smith
@ejsmith
Mar 24 2016 19:43
yeah, which is totally fine with me. :-)
Sander Rijken
@srijken
Mar 24 2016 19:43
yeah indeed, it’s not helpful at all
Eric J. Smith
@ejsmith
Mar 24 2016 19:44
yeah, we could show the value there.
Sander Rijken
@srijken
Mar 24 2016 19:44
also marking them fixed/hidden will be way faster
Eric J. Smith
@ejsmith
Mar 24 2016 19:44
so here is the deal, we have value… it’s supposed to be this generic thing where I can use it however I want...
Sander Rijken
@srijken
Mar 24 2016 19:48
yeah..
one think I can think of that would cause a problem is when value doesn’t contain something that can be aggregated using SUM
does that make sense?
Blake Niemyjski
@niemyjski
Mar 24 2016 20:07
yeah
doesn’t make sense to store deduped count in value even though we want to cause it can be any value which means it cant acurately represent deduped count
plus side is in our formatter if it’s a known field we can put a (5) with a hover and say five additional occurances
Sander Rijken
@srijken
Mar 24 2016 20:09
yeah
Sander Rijken
@srijken
Mar 24 2016 20:21
the bad news for that is that the dedupe work is stuck until that’s done as well
Blake Niemyjski
@niemyjski
Mar 24 2016 20:24
it woudn’t take long to update this stuff I don’t think
except for term stats would be off on total acounts
counts*
and all timelines
sigh
easy enough to fix
just couldn’t rely on total doc counts that elastic gives us :(
be an extra query everywhere
and then confusion when you look at your usage and see 1000 total events and dashboards are showing you 15000 events
Blake Niemyjski
@niemyjski
Mar 24 2016 20:35
unreal
just thinik I found a way for us to shave a ton of time off our queries
like in half
so 2 issues I just found while looking into how we could add affected users
when populating project list we are doing term queries which is fine.. but we are also doing a cache look up of total project count by org id
but it was getting docs and then count.. which we never do that call anywhere so we were storing way more than we needed.
I see there is a foundatio method for doing this.. but I need to ensure the cache key is being cleared.. I don’t think it’s being invalidated.
then the big reveal..
we are populating timeline stats per term query look up for all terms
and not using it anywhere
that’s a lot of extra work
Think I’m going to make term stats generic like we have the numbers thing..
that way we can populate total from a query if we need to
but just trying to dig deeper
cause seems like for these term queries...
for stack lists we only want count… for stack page we want everything for project list we use one thing but were running all the queries all the time
Sander Rijken
@srijken
Mar 24 2016 20:43
not following some of that..
Eric J. Smith
@ejsmith
Mar 24 2016 20:54
reading...
Blake Niemyjski
@niemyjski
Mar 24 2016 21:06
basically we were doing work to calculate fields that were never used as well as timeline data that was never used on every request
I’m making a generic version like our stats stuff that you pass it a list of what you want
Eric J. Smith
@ejsmith
Mar 24 2016 21:06
working with someone on something… I am going to think about the value thing tonight and we will make a decision.
Blake Niemyjski
@niemyjski
Mar 24 2016 21:06
and it runs a query with only that
ok
feel free to call me
Blake Niemyjski
@niemyjski
Mar 24 2016 21:49
against 1 index of prod data… with the new term stats query (same data) minus stats and 2 extra aggregations (min and max date on the outside which was never being used)
shaved off about .6 seconds
:D
the whole site is about to get a whole lot faster
kinda excited about this
this is filtered to the exceptionless project and organization for getting terms by project and stack :)
Blake Niemyjski
@niemyjski
Mar 24 2016 22:25
@ejsmith found a bug with foundatio repositories
confirmed my suspicion
if you call count with a cache key
count-: was the prefix.. I changed it to be :count as the suffix…
but question is should we even add a suffix.. kind of there responsibility
then for cached page results and counts are NOT being invalidated
Eric J. Smith
@ejsmith
Mar 24 2016 22:26
I don’t want suffixes
I want prefixes… namespaced ketys
Blake Niemyjski
@niemyjski
Mar 24 2016 22:26
almost like the invalidate cache should do a remove on the key which I think would clear the child entries : (anything suffixed)
ok
but if you do company:id as the cache key
I’d expect the cache key to stay the same or be company:id:count
how do I invalidate the cache if we are prefixing things?
if you delete id from the cache does that remove keys in a child namespace in redis?
Eric J. Smith
@ejsmith
Mar 24 2016 22:30
no
redis knows nothing about those… its just display.
Blake Niemyjski
@niemyjski
Mar 24 2016 22:31
so why would we want a prefix on these when we could do an intelligent cache clear on the key name and anything under it?
with a delete by prefix
count was the only one specifying a prefix for the cache key
Eric J. Smith
@ejsmith
Mar 24 2016 22:32
yeah, didn’t know where this was.
Blake Niemyjski
@niemyjski
Mar 24 2016 22:34
dang.. can’t really optimize the remove by prefix
I think that’s what we should be doing don’t you agree?
they are all scoped + doc id
part that sucks is we just turned 1 call into 20 for redis
if we did it
but it ensures paged data and counts are cleared
Blake Niemyjski
@niemyjski
Mar 24 2016 22:41
crap
that won’t work
:(
would work for getbyid
countbyid
but nothing else
Eric J. Smith
@ejsmith
Mar 24 2016 22:44
might not be able to cache them
Blake Niemyjski
@niemyjski
Mar 24 2016 22:46
    public Task<long> GetCountByCompanyAsync(string company) {
        return CountAsync(new CompanyQuery().WithCompany(company).WithCacheKey(company));
    }

    protected override async Task InvalidateCacheAsync(ICollection<ModifiedDocument<Employee>> documents) {
        if (!IsCacheEnabled)
            return;

        var options = Options as IQueryOptions;
        if (documents != null && documents.Count > 0 && options != null && options.HasIdentity) {
            var keys = documents.Select(d => $"{d.Value.CompanyId}:count").Distinct().ToList();

            if (keys.Count > 0)
                await Cache.RemoveAllAsync(keys);
        }

        await base.InvalidateCacheAsync(documents);
    }
all of that just for that
exceptionless/Foundatio.Repositories@5b9c822
guess I can make that a prefix
hmm
company:count:companyid
company:companyId:count
I like the later..
but guess all depends on your cache key
could have a bunch of stuff being cached not by a companyid
I reverted back to prefix
Eric J. Smith
@ejsmith
Mar 24 2016 22:55
ok
Eric J. Smith
@ejsmith
Mar 24 2016 23:01
cache tagging would be nice, eh?
Blake Niemyjski
@niemyjski
Mar 24 2016 23:06
Yeah
Real nice
Eric J. Smith
@ejsmith
Mar 24 2016 23:06
too bad it won’t work on cluster
Blake Niemyjski
@niemyjski
Mar 24 2016 23:07
Be nice if we even stored a list in redis of the id
With all the entries and the delete all the keys in the list
Eric J. Smith
@ejsmith
Mar 24 2016 23:11
yeah, seems like there should be a way, but everything I have read says its not going to work later… but neither is the delete by prefix
and we are doing that now