These are chat archives for ManageIQ/manageiq/performance

17th
May 2016
Alex Krzos
@akrzos
May 17 2016 14:06
anyone see this before:
[----] W, [2016-05-17T10:05:53.887753 #31811:a65e8c] WARN -- : Session::DalliStore#set: Value too large, memcached can only store 1048576 bytes per key [key: MIQ:VMDB:531c608fac97e7b756a4d4c6b057cce4, size: 1651151]
Keenan Brock
@kbrock
May 17 2016 14:06
yea - UI room may have a better idea
never seen it, but heard of it
Alex Krzos
@akrzos
May 17 2016 14:07
I can't add a rbac group in the ui
it keeps kicking me out with
Keenan Brock
@kbrock
May 17 2016 14:08
@chrisarcand ^
Joe Rafaniello
@jrafanie
May 17 2016 14:18
@akrzos I'd assume it's logging some warnings in the evm.log
Alex Krzos
@akrzos
May 17 2016 14:19
oh yeah
bunch of gold in there
Joe Rafaniello
@jrafanie
May 17 2016 14:19
that seems like a slow thing to do
Joe Rafaniello
@jrafanie
May 17 2016 14:20
@kbrock not pointing fingers, but I figure you may have seen this and have opinions ;-)
Jason Frey
@Fryguy
May 17 2016 14:21
@jrafanie It dumps the size, but not the contents in prod (I think)
Joe Rafaniello
@jrafanie
May 17 2016 14:22
it still needs to calculate the size on each request?
Jason Frey
@Fryguy
May 17 2016 14:22
if it's going to print it, it does :)
however, lines 2070-2081 are blowing my mind
because it recurses even in prod
Keenan Brock
@kbrock
May 17 2016 14:23
that is sketchy in development mode...
Joe Rafaniello
@jrafanie
May 17 2016 14:23
Yeah, I guess, do we actively get evm.log from production mode users?
seems like wasted cycles in production mode
Keenan Brock
@kbrock
May 17 2016 14:24
seems like wasted cycles in development ;)
Jason Frey
@Fryguy
May 17 2016 14:24
dev seems fine...how else would we find these large objects...
Joe Rafaniello
@jrafanie
May 17 2016 14:24
maybe an opt-in value would be useful for QE or power users
Jason Frey
@Fryguy
May 17 2016 14:24
I mean Alex actually found something...sounds like it worked :D
Keenan Brock
@kbrock
May 17 2016 14:24
heh
Joe Rafaniello
@jrafanie
May 17 2016 14:24
yeah, Alex qualifies as a power user with an opt-in value
;-)
Jason Frey
@Fryguy
May 17 2016 14:25
yeah, I agree @jrafanie ... prod seems useless
The other way is just make it debug in prod, and info in dev
or warn in dev
then do a $log.debug {}
Joe Rafaniello
@jrafanie
May 17 2016 14:29
I don't know how expensive this thing is but I wonder if it's better to rescue the memcache error and only do it there
or additionally, do it all the time in dev or with some flag set
Jason Frey
@Fryguy
May 17 2016 14:31
I think if we get the memcache error we should definitely print it out regardless of prod/dev
but other times, when it's smaller, I think dev only is fine
Joe Rafaniello
@jrafanie
May 17 2016 14:32
@akrzos, can you open an issue? I'm guessing we're storing AR objects in session in the trees... I'm not sure what changed, maybe you know?
vat_tree <String> Size 239104
myco_tree <ActiveSupport::SafeBuffer> Size 656288
hac_tree <String> Size 735579
Keenan Brock
@kbrock
May 17 2016 14:32
@Fryguy so you said to add a catch and throw this in there
Alex Krzos
@akrzos
May 17 2016 14:34
@jrafanie Roger that I'll open an bz with production and evm logs on 5.4 and I'll see if it's the same on 5.5
Chris Arcand
@chrisarcand
May 17 2016 14:38
@akrzos Catching up, not sure if this is related to the group issue ^ Yes, please open a BZ or some issue with more details :+1:
@chrisarcand FYI
Running through 5.4 with a patch, then on to 5.5
Chris Arcand
@chrisarcand
May 17 2016 14:57
Oh, this is a UI bug in 5.4. I thought this was an issue with tenancy @kbrock
weird one. Thanks @akrzos
Alex Krzos
@akrzos
May 17 2016 14:58
@chrisarcand I believe it applies to 5.5 as well
Chris Arcand
@chrisarcand
May 17 2016 14:59
k
Alex Krzos
@akrzos
May 17 2016 14:59
I'll checking right now , but similiar to 5.4 i can't seem to add a group in 5.5 for this database
Joe Rafaniello
@jrafanie
May 17 2016 15:49
@akrzos Is it possible for you to test if calculating the session size is expensive in production with medium/large databases?
I'd think you could do a rough check by logging before/after this line
Alex Krzos
@akrzos
May 17 2016 15:50
I guess I'm not entirely sure I understand what exactly i'd be checking
Joe Rafaniello
@jrafanie
May 17 2016 15:51
Sorry, this line
Alex Krzos
@akrzos
May 17 2016 15:51
so the session size changes from database to database?
Keenan Brock
@kbrock
May 17 2016 15:51
@chrisarcand cool. now that you mention it - it is back in my brain. I knew I should ping you, just couldn't figure out why.
Joe Rafaniello
@jrafanie
May 17 2016 15:51
Yeah, I'd imagine it depends greatly by what you navigate through in the UI and what is drawn in the trees
Keenan Brock
@kbrock
May 17 2016 15:52
there is a lot of work done from the ui team from 5.4 to 5.6 to remove a bunch of the stuff in the session
Joe Rafaniello
@jrafanie
May 17 2016 15:52
Or just comment out that line and compare visits to the same pages before/after commenting it out as measured by the production.log
Alex Krzos
@akrzos
May 17 2016 15:59
@jrafanie ok so We can take a look on the two databases i have and see the size of the sessions and determine how they grow as more of the UI is navigated
Joe Rafaniello
@jrafanie
May 17 2016 16:03
@akrzos Sorry, I was more concerned with how expensive(time/memory) it is to calculate this size as you navigate around. If it's negligible, it's fine the way it is.
I'd imagine the rest api calls do this calculation too
@akrzos DM me the location of the db and I can quickly measure if it's worth changing the way we do this
Alex Krzos
@akrzos
May 17 2016 16:09
OK I PM-ed the ips
I intend on continuing to the use the appliance so I'd prefer no modifications that can't be un-done, I can provide the actual database dumps too but that does take a bit of time to get setup so probably fastest for you to test right on the appliance
This message was deleted
Alex Krzos
@akrzos
May 17 2016 17:25
So did some patches aim to reduce memory on 5.5.4.1?
Jason Frey
@Fryguy
May 17 2016 17:38
I think @kbrock had speed fixes particularly in the UI... Not sure if any memory changes were intentional/side effect/trade-off
@kbrock was the save inventory :_object stuff in 5.5? If so we might want @durandom 's change backported cc @blomquisg
Greg Blomquist
@blomquisg
May 17 2016 17:40
heh,I was gonna say "It is backported!" (to darga)
Keenan Brock
@kbrock
May 17 2016 18:43
no, the :_object stuff was from ManageIQ/manageiq#4532
but it was probably cut and paste code there
Jason Frey
@Fryguy
May 17 2016 18:45
that's for VMs, but in general
oh wait, was the whole thing added there? I thought you had added it originally
Keenan Brock
@kbrock
May 17 2016 18:46
@akrzos in general, I moved a bunch of logic into sql so we wouldn't load vm records into memory
e.g.: not sorting in memory - not sure when that went in
Jason Frey
@Fryguy
May 17 2016 18:46
I'm asking specifically about :_object
Keenan Brock
@kbrock
May 17 2016 18:46
I think that pr introduced it
Jason Frey
@Fryguy
May 17 2016 18:47
oh wow ok...I was totally off :)
Keenan Brock
@kbrock
May 17 2016 18:47
I had wanted to do someting like that for a while, but never found a good way to do it
whether automatically adding the :id or the :_object