These are chat archives for ManageIQ/manageiq/performance

1st
Dec 2017
Dennis Metzger
@dmetzger57
Dec 01 2017 15:57
@jrafanie ran a test comparing “MALLOC_MMAPTHRESHOLD" default value against 8192, here’s what that showed (not an improvement):
malloc_mmap.pdf
there’s an ‘_’ on that variable :smile:
Joe Rafaniello
@jrafanie
Dec 01 2017 16:07
cool @dmetzger57, do we have logs with the GC stat information from NIckL's initializer?
Dennis Metzger
@dmetzger57
Dec 01 2017 16:15
That’s a question for @NickLaMuro, PTO yesterday/today for me with limited connectivity
Joe Rafaniello
@jrafanie
Dec 01 2017 16:16
Oh, ok. Have a good time off then @dmetzger57. Thanks for the update!
Nick LaMuro
@NickLaMuro
Dec 01 2017 16:20
info sent in a PM
Joe Rafaniello
@jrafanie
Dec 01 2017 16:20
Thanks @NickLaMuro
Joe Rafaniello
@jrafanie
Dec 01 2017 17:12
@NickLaMuro here's what I'm seeing on your appliance in the Server process (13164)...
in a span of 32 hours (if I mathed correctly), RSS went up 210 MB, threads went DOWN (35 to ~32), heap live slots is very close...
[----] I, [2017-11-30T03:07:40.902684 #13164:a2f134]  INFO -- : MIQ(MiqServer#monitor_poll) Memory Info XXXX => {RSS: 612400000, PSS: 451880000, thread_count: 35, gc_stats: {:count=>3669, :heap_allocated_pages=>4746, :heap_sorted_length=>5168, :heap_allocatable_pages=>240, :heap_available_slots=>1934473, :heap_live_slots=>1686603, :heap_free_slots=>247870, :heap_final_slots=>0, :heap_marked_slots=>1333944, :heap_swept_slots=>262342, :heap_eden_pages=>4492, :heap_tomb_pages=>254, :total_allocated_pages=>4746, :total_freed_pages=>0, :total_allocated_objects=>1140769415, :total_freed_objects=>1139082812, :malloc_increase_bytes=>26554152, :malloc_increase_bytes_limit=>33554432, :minor_gc_count=>3467, :major_gc_count=>202, :remembered_wb_unprotected_objects=>54860, :remembered_wb_unprotected_objects_limit=>109690, :old_objects=>1236534, :old_objects_limit=>2472634, :oldmalloc_increase_bytes=>58780320, :oldmalloc_increase_bytes_limit=>91630505}, objspace_count: {:TOTAL=>1934473, :FREE=>247817, :T_OBJECT=>158425, :T_CLASS=>19039, :T_MODULE=>2631, :T_FLOAT=>12, :T_STRING=>655861, :T_REGEXP=>2458, :T_ARRAY=>244182, :T_HASH=>77288, :T_STRUCT=>13565, :T_BIGNUM=>173, :T_FILE=>187, :T_DATA=>90520, :T_MATCH=>32985, :T_COMPLEX=>1, :T_RATIONAL=>2218, :T_SYMBOL=>3640, :T_IMEMO=>376359, :T_ICLASS=>7112}}
...
[----] I, [2017-12-01T11:47:47.536950 #13164:a2f134]  INFO -- : MIQ(MiqServer#monitor_poll) Memory Info XXXX => {RSS: 822052000, PSS: 566237000, thread_count: 32, gc_stats: {:count=>14094, :heap_allocated_pages=>4746, :heap_sorted_length=>5168, :heap_allocatable_pages=>131, :heap_available_slots=>1934473, :heap_live_slots=>1715793, :heap_free_slots=>218680, :heap_final_slots=>0, :heap_marked_slots=>1337475, :heap_swept_slots=>251161, :heap_eden_pages=>4569, :heap_tomb_pages=>177, :total_allocated_pages=>4746, :total_freed_pages=>0, :total_allocated_objects=>4308932117, :total_freed_objects=>4307216324, :malloc_increase_bytes=>29995568, :malloc_increase_bytes_limit=>33554432, :minor_gc_count=>13250, :major_gc_count=>844, :remembered_wb_unprotected_objects=>55036, :remembered_wb_unprotected_objects_limit=>109670, :old_objects=>1239850, :old_objects_limit=>2471368, :oldmalloc_increase_bytes=>63798280, :oldmalloc_increase_bytes_limit=>96253255}, objspace_count: {:TOTAL=>1934473, :FREE=>218627, :T_OBJECT=>162905, :T_CLASS=>19170, :T_MODULE=>2631, :T_FLOAT=>12, :T_STRING=>663119, :T_REGEXP=>2458, :T_ARRAY=>253077, :T_HASH=>79385, :T_STRUCT=>13688, :T_BIGNUM=>207, :T_FILE=>187, :T_DATA=>94557, :T_MATCH=>33384, :T_COMPLEX=>1, :T_RATIONAL=>2844, :T_SYMBOL=>3640, :T_IMEMO=>377469, :T_ICLASS=>7112}}
total allocated pages is the same
Nick LaMuro
@NickLaMuro
Dec 01 2017 17:17
could a c-extension be leaking memory then?
Joe Rafaniello
@jrafanie
Dec 01 2017 17:17
good question
I wonder if that would show up as a non-ruby heap file in the /proc/pid/smaps file
Jason Frey
@Fryguy
Dec 01 2017 17:30
c-extension def seems possible
could possibly also be a core ruby leak
Joe Rafaniello
@jrafanie
Dec 01 2017 17:36
Would those show up in the ruby heap or off heap?
Joe Rafaniello
@jrafanie
Dec 01 2017 17:41
there's 132 "files" in smaps > 2000 kB
the top ones are:
832988 kB
103588 kB
8188 kB
4100 kB
those are...
01428000-34441000 rw-p 00000000 00:00 0                                  [heap]
Size:             835684 kB
7f175d312000-7f176383b000 r--p 00000000 fd:00 12590537                   /usr/lib/locale/locale-archive
Size:             103588 kB
7ffd9741d000-7ffd97c1c000 rw-p 00000000 00:00 0                          [stack]
Size:               8188 kB
7f1752818000-7f1752c19000 rw-p 00000000 00:00 0
Size:               4100 kB
Joe Rafaniello
@jrafanie
Dec 01 2017 17:50
it appears to be mostly the ruby heap although i don't know why that 100 mb locale-archive is so big
01428000-34441000 rw-p 00000000 00:00 0                                  [heap]
Size:             835684 kB
Rss:              829460 kB
Pss:              613144 kB
Shared_Clean:          0 kB
Shared_Dirty:     303492 kB
Private_Clean:         0 kB
Private_Dirty:    525968 kB
Referenced:       827272 kB
Anonymous:        829460 kB
AnonHugePages:         0 kB
Swap:               5548 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd
Joe Rafaniello
@jrafanie
Dec 01 2017 17:57
This all seems consistent with malloc doing shenanigans because we're not threading correctly
ruby heap growing in OS land, ruby live objects staying consistent, thread count staying the same (even going down)
Joe Rafaniello
@jrafanie
Dec 01 2017 19:23
/play crickets
Nick LaMuro
@NickLaMuro
Dec 01 2017 19:31
sorry, worry on something else today
Joe Rafaniello
@jrafanie
Dec 01 2017 19:33
;-)
holler if you need help
Nick LaMuro
@NickLaMuro
Dec 01 2017 19:34
well, it is my old friend the mime-types-redirector + automate...
Joe Rafaniello
@jrafanie
Dec 01 2017 19:40
Good times