These are chat archives for ruby-bench/ruby-bench-web

26th
Feb 2015
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:12
Hey Kir!
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:17
@kirs hey@
opps
Kir Shatrov
@kirs
Feb 26 2015 13:44
hey!
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:44
Sorry I was alittle late
Kir Shatrov
@kirs
Feb 26 2015 13:44
me too, sorry
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:45
no worries@
What are your plans for your talk during RailsConf?
mostly about how performance changed few rails major releases
and how the tools can help to prevent performance regressions
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:47
For now RubyBench has releases and commits benchmarks
Kir Shatrov
@kirs
Feb 26 2015 13:47
do you have a hook on ruby-core repo?
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:47
nope
I'm hooking from my fork of ruby
so I manually update it everyday
Kir Shatrov
@kirs
Feb 26 2015 13:47
oh
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:47
Eventually I'll want it to be a hook from Ruby core
Kir Shatrov
@kirs
Feb 26 2015 13:47
so for Railsperf I have a hook DHH set
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:48
Just that we're still in development so I don't want the hooks to be flooding in every now and then
Kir Shatrov
@kirs
Feb 26 2015 13:48
(but there is no commit-build support yet)
cool
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:48
Is Rails looking for something like Travis?
if the benchmark build fails we wont merge?
Kir Shatrov
@kirs
Feb 26 2015 13:49
kind of. The best case for me is to build every PR and report performance with the bot
at the PR comments
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:50
Yea it can be done.
just have to figure out how.
Ruby is still using a seperate issue tracker so I didn't focus on that.
Currently it is just a tool which we use to track and report if we find a regression.
Kir Shatrov
@kirs
Feb 26 2015 13:51
the biggest issue I faced: cloning huge rails repo and bundling all dependencies (https://github.com/rails/rails/blob/master/Gemfile) inside docker takes up to 20 min
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:51
I'm open to suggestions how it can be done since I've never implemented something like that before
Kir Shatrov
@kirs
Feb 26 2015 13:51
did you set up any cache for docker?
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:52
Yea.. From what I know, you can only cache when building a docker container. Not sure if we can cache things when running scripts inside the container
O
Kir Shatrov
@kirs
Feb 26 2015 13:53
omg
right in the repo?
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:54
yup I copy it into the docker container
because we benchmark against Discourse against a fixed commit
I generated those with bundle package or smth. Can't remember right now
I'm sure we can do something similar
since gems are not updated every day
Kir Shatrov
@kirs
Feb 26 2015 13:55
it makes sense if you benchmark it against a fixed commit
it won't work for rails Gemfile cuz it's updated every week
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:57
If we maintain a cache, gems that aren't affected can still be install from the cache right? not too sure about this until I try it out
Kir Shatrov
@kirs
Feb 26 2015 13:57
yes
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:57
O ok you mean installing the Rails gem?
Kir Shatrov
@kirs
Feb 26 2015 13:58
cache problem affects 2 things
1) cloning rails repo for specific commit
2) bundling all rails dependencies
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:58
we could install the initial Rails repo
and git fetch perhaps?
Kir Shatrov
@kirs
Feb 26 2015 13:59
yeah
you do that for ruby-core?
Guo Xiang Tan
@tgxworld
Feb 26 2015 13:59
yup
every commit I'm just rebasing it against the latest trunk
so I don't have to clone the entire repo
overtime it'll slow down but we could rebuild the container once a week automatically in the future
Kir Shatrov
@kirs
Feb 26 2015 14:00
cool
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:01
some of my design decisions may be wrong though but we can change them XD
Kir Shatrov
@kirs
Feb 26 2015 14:01
then I can experiment with creating rails directory in https://github.com/ruby-bench/ruby-bench-docker and creating the runner for trunk rails
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:01
yea
Kir Shatrov
@kirs
Feb 26 2015 14:02
what next? do you run scripts from ruby-bench-docker repo by schedule?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:02
we can speed up the time taken to benchmark as we go along
nope everyday I pull in all the latest commits
we use a GitHub hook to trigger the script
scripts
Kir Shatrov
@kirs
Feb 26 2015 14:04
github hook on your ruby fork?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:04
yup
everytime I push the latest commits to my fork it'll trigger the hook and push all the latest commits to my ruby-bench-web
Kir Shatrov
@kirs
Feb 26 2015 14:05
got it
Kir Shatrov
@kirs
Feb 26 2015 14:06
so after I create the docker runner for rails we just add the repo to ruby-bench-web and set up the hook?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:06
So I have a class that stores the necessary commands to run the scripts. I feel like it couples ruby-bench-docker and ruby-bench-web but I've not really worked on decoupling it yet
I can maintain a fork of the repository first and when the time comes we can easily transfer all the commits to the official Rails repo
Kir Shatrov
@kirs
Feb 26 2015 14:08
sounds like a plan
  1. create the docker container and runner scripts
  2. patch remote_server_job.rb to support rails_trunk runner
  3. pull commits and set up the fork
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:08
Feel free to ask me any questions you have. I'm really happy for RubyBench to merge with RailsPerf :smile:
yea
Kir Shatrov
@kirs
Feb 26 2015 14:09
thanks!
do you also organize red dot ruby?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:10
Nope but I'm joining the organizing committee this year though
Kir Shatrov
@kirs
Feb 26 2015 14:10
congrats!
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:10
Thanks!
wiill you be coming?
Kir Shatrov
@kirs
Feb 26 2015 14:10
my colleague gave a talk there last year
maybe I will
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:11
cool!
do you know if it is possible to do something like ruby rails_benchmark_scripts.rb?
basically the script itself will install all the necessary gems and stuff
we have ruby-bench-script so I think people should be able to run the scripts from there without going through the docker contains
Kir Shatrov
@kirs
Feb 26 2015 14:13
the script will be like
`git clone rails/rails`
`cd rails && bundle`

# now you can run the benchmarks
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:15
yup something like the minimum test scripts that people submit when reporting an issue
Kir Shatrov
@kirs
Feb 26 2015 14:16

we have ruby-bench-script

do you mean the repo?

Guo Xiang Tan
@tgxworld
Feb 26 2015 14:16
yea
the repo
Kir Shatrov
@kirs
Feb 26 2015 14:16
can't find it
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:17
opps sorry
suite
ruby-bench-suite
Kir Shatrov
@kirs
Feb 26 2015 14:19
but it would duplicate the Docker scripts/runner, isn't it?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:20
o the idea is to pull the scripts from that repo
so when we update the scripts we don't have to rebuild the containers
Kir Shatrov
@kirs
Feb 26 2015 14:20
that's possible
how long is the build time now?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:20
o building the container is fine. We actually build it once and just pull it into the server
any changes into the repo is fetch using git in the runner script
Kir Shatrov
@kirs
Feb 26 2015 14:22
but when you add new benchmark script, you will need to update the container?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:23
unless the way we run the benchmark script changes
then we'll have to rebuild
Kir Shatrov
@kirs
Feb 26 2015 14:31
how do the benchmarks report metrics back to web app?
not sure if it is the best way but currently
through a web hook
Kir Shatrov
@kirs
Feb 26 2015 14:33
oh in railsperf I used another way
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:34
o how did you do it?
one thing is that the scripts are run on another server
so we have to send the results back
Kir Shatrov
@kirs
Feb 26 2015 14:34
when job executes docker run ... it collects the output
and output contains json metrics
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:35
ah icic
you print it out?
yes
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:35
one of the problems I have is that Heroku requires an active worker when running Jobs
doesn't really work well with long running jobs because the workers is restarted everytime we depoly
I'm open to how we can change it do
patching the driver feels like a hack to me
Kir Shatrov
@kirs
Feb 26 2015 14:37
I'd avoid sidekiq for long jobs
but it's still an issue on heroku
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:37
using Delayed::Job right now :P
Kir Shatrov
@kirs
Feb 26 2015 14:37
shit
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:37
DelayedJob
is it bad?
Kir Shatrov
@kirs
Feb 26 2015 14:38
well it's ok for small amount of jobs
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:38
I'm using ActiveJob so switching should be fine
Kir Shatrov
@kirs
Feb 26 2015 14:39
cool
don't you think Rails of overkill for such an app?
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:39
Hehe not too sure. I'm quite junior as a dev so I know Rails the best
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:40
interesting. I've never looked into sinatra much though
Kir Shatrov
@kirs
Feb 26 2015 14:47
I just don't like having huge Rails structure for apps with only few routes
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:48
ic
ok I'm heading off now! email me if you have any questions!
I'll be happy to answer them :)
Kir Shatrov
@kirs
Feb 26 2015 14:49
thanks!
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:49
It was really nice chatting with you
Kir Shatrov
@kirs
Feb 26 2015 14:50
pleasure to chat with you
Guo Xiang Tan
@tgxworld
Feb 26 2015 14:50
Let me know if you need any help too!
Kir Shatrov
@kirs
Feb 26 2015 14:50
sure