These are chat archives for ManageIQ/manageiq/performance

11th
Apr 2018
Pradeep Kumar Surisetty
@psuriset
Apr 11 2018 01:11
@kbrock agree . 50 mins is justfied, since we do it for other objects. I am not sure about hawkular, if it can handle 5x data at a time.
Keenan Brock
@kbrock
Apr 11 2018 14:51
@NickLaMuro I was looking at http://engineering.appfolio.com/appfolio-engineering/2018/4/2/rails-ruby-bench-what-and-why which spurred my questions of how we could use the existing benchmark infrastructure
Nick LaMuro
@NickLaMuro
Apr 11 2018 15:12
which questions were those?
(also, reading...)
Keenan Brock
@kbrock
Apr 11 2018 15:13
if we could use an existing benchmark that we didn't write ourselves to test our software
ruby benchmark has gotten much better documentation e.g.: https://github.com/ruby-bench/ruby-bench#contributing
maybe that is just our base line for running/storing/comparing.
We still need to write the driver (e.g.: [memory_driver], [driver]) that we will use to determine the values
NickLaMuro @NickLaMuro kinda wants to watch the talk referenced in that article now...
Nick LaMuro
@NickLaMuro
Apr 11 2018 15:24

if we could use an existing benchmark that we didn't write ourselves to test our software

Honestly, I would have to do more than read that article to answer that question

Nick LaMuro
@NickLaMuro
Apr 11 2018 17:36

@kbrock After taking some time to look through this, I don't think there is a lot we can reuse:

  • From Noah's Repo, the main components to the actually benchmark, user_simulator.rb and seed_db_data.rb are all custom to Discord and their probably isn't a whole lot that we can carry over
  • As far as what data he is collecting, it is basically wall clock time to run a request. Nothing to do with number of queries, memory used, cpu time, etc. This is fine for what he is trying to test, but I feel that it might be limiting to what you (and I) would like
  • The scripts (packerbeing the tool of choice in this case) used to build the framework are specifically around configuring for the things he is testing, specifically, against Discord on ubuntu.

There are other pitfalls, but I know for certain that it won't be a gem install rails-ruby-bench and into manageiq and we would get a free benchmarking suite against our application. The project is just not designed with that in mind.


That said, there is a LOT we can learn from in it and emulate, and things I have done myself in manageiq-performance (though, not documented :sweat_smile: ...) to simulate a lot of the above and could be improved to better match that. For example, there is a lot of functionality in lib/manageiq_performance/requestor.rb that is specifically meant to do the parts of user_simulator.rb in rails_ruby_bench, and could definitely use some love to make better.

Jason Frey
@Fryguy
Apr 11 2018 17:44
@NickLaMuro At some point you should write a blog post about manageiq-performance and the myriad features :)
Nick LaMuro
@NickLaMuro
Apr 11 2018 18:55
Definitely cool with the idea, but I think having actually docs in place would allow me to reference to them so I don't get too wordy in the post itself... not that I am a wordy person in general, just saying...
Keenan Brock
@kbrock
Apr 11 2018 18:55
@NickLaMuro ok, I like this driver: [rails/benchmarks/driver.rb] -- it records 2 numbers, including wall time and number of objects allocated. visible as a comparison here
you know I don't like wall time. I am looking for something that measures memory (some tests measure memory) number of queries (haven't seen one of these) and other stuff. wall time is too ambigious temperamental (but I accept others like it)
Keenan Brock
@kbrock
Apr 11 2018 19:23
@NickLaMuro each suite of tests collects different numbers because different numbers make sense in different contexts
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:25
i understand. I wasn't trying to suggest "wall clock is bad, mmkay", just that in the specific case of rails_ruby_bench, that is the only telemetry data he is collecting
Keenan Brock
@kbrock
Apr 11 2018 19:25
heh. no, I was saying that wallclock is bad (or its for that matter)
and b. depending upon the test suite, they are collecting other stuff too
I'm not impressed how each suite has a different driver. how big the drivers are. and how the code location is hardcoded in the script. seem that could be a docker variable or something but...
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:27

are you still commenting on https://github.com/noahgibbs/rails_ruby_bench , or https://github.com/ruby-bench/ruby-bench-suite

Because I haven't had the time to look at the latter, so I seriously have "no comment" on it (yet)

Keenan Brock
@kbrock
Apr 11 2018 19:27
also, they run on raw hardware, so a little odd how they use use docker
sorry - I'm talking ruby-bench
isn't noah gibbs based upon ruby-bench?
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:27
no, see the URL
and the talk that he linked in the article that you posted
Keenan Brock
@kbrock
Apr 11 2018 19:28
yes that I posted
the web is so tangled. confuses me
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:29
I just need to use this as my periodic reminder that @kbrock is a "serial skim-reader"
correction: 'most people on the planet are "serial skim-readers"' #aintNobodyGotTimeForDat
Keenan Brock
@kbrock
Apr 11 2018 19:30
you can lump me in with them. or call me an outlier - both work
huh
ruby bench is so big - why come up with yet another benchmark?
both are benchmarking discourse too
both use multiple ruby versions
sigh
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:31
He did give a 30min talk on it at Kaigi2017 that did explain (in part) just that question right there...
Keenan Brock
@kbrock
Apr 11 2018 19:31
thanks
I'll look at my links
Nick LaMuro
@NickLaMuro
Apr 11 2018 19:31
linked in the post you posted...
Keenan Brock
@kbrock
Apr 11 2018 19:43
interesting. rails_ruby_bench has more stars than ruby-bench. I'm surprised by this
Keenan Brock
@kbrock
Apr 11 2018 20:01
thanks @NickLaMuro I watched the rubikaji talk and having read some of noah's posts on discourse it makes more sense
I like the ruby bench display docker image and some of the drivers that run scripts and collect various metrics. think I'll dive into that more