These are chat archives for ManageIQ/manageiq/performance

4th
Jan 2018
Keenan Brock
@kbrock
Jan 04 2018 02:56
yea - was wondering about the second one - it converts central time to eastern
re: eval, lol - I looked at the code and found my own answer
forgot how much you like to put a hash into the log so you can eval it
Nick LaMuro
@NickLaMuro
Jan 04 2018 03:26
well, just makes the parse code a bit simpler, but wouldn't say I "like it"
Keenan Brock
@kbrock
Jan 04 2018 03:26
using it in more than 3 places == like ;)
Nick LaMuro
@NickLaMuro
Jan 04 2018 03:31
fair enough
Beni Cherniavsky-Paskin
@cben
Jan 04 2018 11:30
eval in log parsers is slightly risky, if people might run them on logs provided by others :) Seems ruby has nothing builtin like python's ast.literal_eval() but now found https://github.com/bibstha/safe_parser — probably much slower though...
perhaps we should log with .to_json instead of .to_s.
Keenan Brock
@kbrock
Jan 04 2018 15:28
yea, 'risky' but - nick lives on the edge
Nick Johnny Dangerously LaMuro
Nick LaMuro
@NickLaMuro
Jan 04 2018 15:35
@kbrock the reason I said I didn't "like it" was @cben 's reason. That said, this was never something that I was going to push for in prod, so that is why I didn't really care in this case since it made for a simpler regexp
Keenan Brock
@kbrock
Jan 04 2018 15:46
well, after reading all about flame throwers, I go back to Nick Johnny Dangerously LaMuro
https://what-if.xkcd.com/130/
Nick LaMuro
@NickLaMuro
Jan 04 2018 15:47
did you just repost something I posted...
Keenan Brock
@kbrock
Jan 04 2018 16:17
oh yea
that is why I posted it
saying if you posted that.... then that just shows you are Mr. Dangerous
(well, danger is your middle name)
Nick LaMuro
@NickLaMuro
Jan 04 2018 16:20
and I have two first names
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:04
don't Austen Powers movie quote me when you were clearly referencing "Johnny Dangerously"
Keenan Brock
@kbrock
Jan 04 2018 17:13
yea
I realized that I messed up
Johnny Dangerously isn't his middle name
I screwed that up
but danerous is austin power's middle name
wasn't sure how to recover from that
so sorry to offend
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:15
you take my snark way to seriously...
Keenan Brock
@kbrock
Jan 04 2018 17:15
lol
either that or I take my own jokes way too seriously
Keenan Brock
@kbrock
Jan 04 2018 17:24

If I can remove

.map { |time| (Time.at(time).utc + -14400).strftime("%Y-%m-%dT%H:%M:%S 2000000000") }

then I can merge the date and time, and then the output could be 100% based upon the regex

(ignoring the eval - but we'll leave that for another time)
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:28
@kbrock you gotta link to me what you are referencing... you are clearly now talking about some code that I am unaware of
sorry, nevermind, found it
sorry
out of context (as usual) :(
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:36
@kbrock I am confused... that whole section is specific to the fact that I am doing a directory scan for memory dump files based on the pid currently being processed, and so the entire block of code there could probably just go away if you are trying to make this more generic

basically it is taking the dumps that have been generated here (and that I would just gzip manually):

https://github.com/NickLaMuro/miq_tools/blob/5749a3e3add2e05cac106eb5bc78bc55cd0bc53c/memory_leak_worker_logging/patch.diff#L41

And creating an .impulse file to be optionally used in when generating the graph

Keenan Brock
@kbrock
Jan 04 2018 17:39
what do you use the impulse for?
I like your use of opt.separator instead of a comment in the codebase
very nice
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:41
20171026_16170.png
Keenan Brock
@kbrock
Jan 04 2018 17:41
nice
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:41
@kbrock that is what the impulses would have been used for in that case
but it was an option arg in the .gnup script
Keenan Brock
@kbrock
Jan 04 2018 17:43
ok, my idea
if you ignore the whole eval thing
if you could pass in a regex, with each capture representing a column to be used in the output
that would allow us to generate various gnu plot graphs without changing too much code
not sure if necessary - we probably always want to output the same graphs
aah - for this script, I tend to pass in a wildcard - having bash determine which files to pass in. could that work instead?
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:49
I mean, that is usually what I do, and how it determines the proper order:
$ ruby ../miq_tools/miq_top_output_processor/top_processor.rb --pid 22782 --worker-type "MIQ Server" tmp/mem_prof_logs/10-[REDACTED]/top_output.log-2018010[3,4,5]*
$ TITLE="MiqServer#monitor\_workers with only Generic, Priority, Schedule wkrs w/o processed\_worker\_ids & sync\_workers" ../miq_tools/miq_top_output_processor/top_info_plot.gnup top_outputs/20180102_22782.data
$ open 20180102_22782.png
ok - just a failsafe - sounds good
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:52
@kbrock so since I started using multiple boxes, I started using the IP as part of the directory structure to separate the two
see:
tmp/mem_prof_logs/10-[REDACTED]/top_output.log-2018010[3,4,5]*
Keenan Brock
@kbrock
Jan 04 2018 17:52
10 = ip?
aah, the 10 is an ip.
ok
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:53
10-10-10-10 for example... not going to paste a real IP in here
Keenan Brock
@kbrock
Jan 04 2018 17:54
gotcha
when I saw the 10- I didn't add 1+1
"10 = ip?" wasn't a challenge - it was an understanding
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:54
saw it as a question
Keenan Brock
@kbrock
Jan 04 2018 17:55
lol. the ? may have suggested that :(
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:55
to be fair, the confusion is valid, it wasn't clear
Keenan Brock
@kbrock
Jan 04 2018 17:55
;)
you merge mutiple input files into a single output file?
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:57
yes, since I am usually looking at multiple days worth of data, and it get's log rotated nightly
Keenan Brock
@kbrock
Jan 04 2018 17:57
nice
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:57
also why I uncompress in ruby, instead of manually, to save on my machine's own disk space
Keenan Brock
@kbrock
Jan 04 2018 17:57
had thought about producing date1-ip1, date2-ip1, date1-ip2, date2-ip2
I like the merge though
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:58
the .gnup script can split the data for you and name it accordingly, so I figured that would be more useful than splitting the data
the date of the .data file is the first occurrence of it, incase that was not clear
Keenan Brock
@kbrock
Jan 04 2018 17:59
no, that was not clear
nice
Nick LaMuro
@NickLaMuro
Jan 04 2018 17:59
probably could have added that to the README... will do that at some point
Keenan Brock
@kbrock
Jan 04 2018 17:59
I like the comments in the code itself - to understand the code
ok
so I was just thinking of a dumb script to slice up a file by pid - nothing else
then maybe creating a directory for each worker to link to each of the logs for that worker
you got time for a little voice? https://bluejeans.com/4878641577
it is close to lunch for you - so we could timebox until your stomach growls
Keenan Brock
@kbrock
Jan 04 2018 18:05
aside: I really like how the patching worked for me:
no patching to the files
it is my split_logs script that needs all the work.
Keenan Brock
@kbrock
Jan 04 2018 20:07
ok, split logs was easier than I thought - even wrote a crystal version. although since our appliances don't have crystal, went back to ruby
ok, yea. crystal is much faster
Jason Frey
@Fryguy
Jan 04 2018 20:11
the appliance doesn't need crystal as long as you compile it elsewhere
it compiles to a native binary
Keenan Brock
@kbrock
Jan 04 2018 20:12
yea - this is just log parsing and stuff
good point though - thanks
Adam Grare
@agrare
Jan 04 2018 20:15
@Fryguy has a gitter parser that raises an alert if someone says crystal :)
Jason Frey
@Fryguy
Jan 04 2018 20:15
what's that I just got pinged again ;)
Adam Grare
@agrare
Jan 04 2018 20:15
:laughing:
Keenan Brock
@kbrock
Jan 04 2018 20:17
heh
I was trying to write a program that ran in both - too tricky