These are chat archives for opf/openproject

24th
Aug 2015
Vladimir K
@cronacronis
Aug 24 2015 09:46
Hello all!
I have measured API v2, have following results:
[Spray_HTTP_API$(akka://default)]    WORKERS: Checking 200
[Spray_HTTP_API$(akka://default)]    WORKERS: Actors total (a): 16
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks total (t): 200
[Spray_HTTP_API$(akka://default)]    WORKERS: Time (milis): 27478
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks per second (t/s): 7.278550112817527
Does it make sensse? Should it be faster?
I am creating tasks with following
{
                "planning_element": {
                    "subject": "This is Scala created task",
                    "type_id": 1
                }
            }
Markus Kahl
@machisuji
Aug 24 2015 10:06
Heya, how did you run OpenProject for that? rails s? Or in a passenger or unicorn setup? Mind, that unlike the wonderful Scala (being based on Java) Ruby does not have multithreading. :D
Florian Kraft
@floriank
Aug 24 2015 10:07
shamless scala plug right there.
We coud always use JRuby ;) (which does have real multithreading!)
Markus Kahl
@machisuji
Aug 24 2015 10:09
That would be a start.
Then again the OpenProject code is most definitely not thread-safe as it is. Who knows how long it would take to fix that.
Vladimir K
@cronacronis
Aug 24 2015 10:11
@machisuji i have used ubuntu package openproject-ce ;)
@machisuji i assume unicorn based :(
Markus Kahl
@machisuji
Aug 24 2015 10:12
Mh I see, I'm guessing that this will run with multiple processes then.
Vladimir K
@cronacronis
Aug 24 2015 10:13
I have unicorn worker[0] mostly
Markus Kahl
@machisuji
Aug 24 2015 10:13
By the way which tool did you use or did you write that by hand using spray?
Vladimir K
@cronacronis
Aug 24 2015 10:13
Just by hand, can commit the code if you want
Markus Kahl
@machisuji
Aug 24 2015 10:14
thanks, that would be nice. Just curious.
you only have one worker?
Vladimir K
@cronacronis
Aug 24 2015 10:14
@machisuji Where I can set multiple unicorn workers in the settings?
@machisuji looks like - yes
Markus Kahl
@machisuji
Aug 24 2015 10:14
That's a good question. @kgalli do you know by any chance?
@cronacronis have you tried openproject scale 4?
that should give you 4 workers
openproject-ce*
or what ever the executable is called
ah no
it says openproject scale TYPE=NUM in the docs
dunno what's that supposed to mean to be honest, I mean the TYPE
Markus Kahl
@machisuji
Aug 24 2015 10:19
I would expect this to just set the number of workers
maybe you can scale background workers and web workers separately
there also seems to be an ENV var for that

you could try:

openproject-ce config:set WEB_CONCURRENCY=4

Vladimir K
@cronacronis
Aug 24 2015 10:23
will do, and report here
nothing sophisticated there
Markus Kahl
@machisuji
Aug 24 2015 10:51
@cronacronis cheers
why are you still using 2.10.x? :D
Vladimir K
@cronacronis
Aug 24 2015 10:53
@machisuji Just for compatibility with my other stuff :( , you can easily switch to 2.11. Should work :)
Vladimir K
@cronacronis
Aug 24 2015 10:59
@machisuji just don't forget to change build.sbt if you use 2.11
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:01
@cronacronis I think you really made @machisuji happy by using Scala xD
Vladimir K
@cronacronis
Aug 24 2015 11:02
haha :)
Vladimir K
@cronacronis
Aug 24 2015 11:14
Yep, went up
```
[Spray_HTTP_API$(akka://default)]    WORKERS: Actors total (a): 16
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks total (t): 500
[Spray_HTTP_API$(akka://default)]    WORKERS: Time (milis): 26441
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks per second (t/s): 18.910026095836013
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:19
I'd say (pure gut feeling) that ~20 work packages per second is quite a good number. Probably still depends on what you want to achieve and will probably look different for more complex payloads.
Vladimir K
@cronacronis
Aug 24 2015 11:23
I would like to have 1k/second at least :)
Client: 24cores
Server: 16cores, 28GB memory
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:24
I see... May I ask what you are doing?
Vladimir K
@cronacronis
Aug 24 2015 11:24
This message was deleted
This message was deleted
[Spray_HTTP_API$(akka://default)]    WORKERS: Actors total (a): 144
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks total (t): 2000
[Spray_HTTP_API$(akka://default)]    WORKERS: Time (milis): 104973
[Spray_HTTP_API$(akka://default)]    WORKERS: Tasks per second (t/s): 19.052518266601886
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:25
What is your use case? I remember that you already explained to me once, but that time around I did not think you would need that much power...
Vladimir K
@cronacronis
Aug 24 2015 11:25
I am trying to use OpenProject as a middleware gateway for processing worlflow of legal cases
Markus Kahl
@machisuji
Aug 24 2015 11:25
how many workers did you use? I guess you can take 16 then?
Vladimir K
@cronacronis
Aug 24 2015 11:25
now i have put 14 workers
2 for system ;)
nevertheless couldnot breach load of 3.5 :(
Markus Kahl
@machisuji
Aug 24 2015 11:26
but yeah generally speaking OpenProject will probably be slower it being Ruby and all. The competition which shan't be named (which is Java-based) might be faster. ;P
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:26
not sure if they can kick 1k/s, though...
Markus Kahl
@machisuji
Aug 24 2015 11:26
probably not
Vladimir K
@cronacronis
Aug 24 2015 11:27
I see now, that 14 workers really doesn't affect anything, the same as with 4 :(
Markus Kahl
@machisuji
Aug 24 2015 11:27
maybe it's worker workers then :D
I mean there's web workers and there's background workers for sending emails and stuff
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:27
@machisuji I am not much into the details there, but is there something like a connection limit to the database or anything?
Markus Kahl
@machisuji
Aug 24 2015 11:27
how did you increase the number of workers? With the WEB_CONCURRENCY var?
could be @NobodysNightmare but I would imagine it's higher than 4
Vladimir K
@cronacronis
Aug 24 2015 11:28
@machisuji yes, corrent
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:28
@cronacronis Which database are you using? Is it running on the same machine as OpenProject?
Markus Kahl
@machisuji
Aug 24 2015 11:29
and the 14 processes are unicorn slaves, yes?
Vladimir K
@cronacronis
Aug 24 2015 11:29
Did you try to build openproject with MongoDB? Both systems are on the cluster.
@NobodysNightmare standard supply of ce, it is mysqld 5.5
@NobodysNightmare Both systems are on the cluster. So technically separate
I will play around with Akka actors, I just cannot load openproject higher :(
there is a bottleneck somewhere
I aggree @NobodysNightmare is right, it is probably MySQL writes.
I want to try with TokuDB, it might get some performance
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:34
You are starting to use words I have no experience with (MongoDB) or I#ve never heard of (TokuDB). I'd really apprechiate any feedback if you found a bottleneck...
Vladimir K
@cronacronis
Aug 24 2015 11:35
I have never written a code on Ruby :( It will be a challenge. I will keep posting updates
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:36
Regarding MongoDB: As far as I know we never tried that, but then I thought that the typical data we are dealing with is quite classical for SQL-based database and not so much for NoSQL ones... But again: Sadly, I really have low experience with Mongo...
Vladimir K
@cronacronis
Aug 24 2015 11:36
TokuDB is a drop-in replacement for InnoDB can be used as faster version of MySQL
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:36
Ah, I see...
Vladimir K
@cronacronis
Aug 24 2015 11:36
Shows better performance for writes
Oliver G√ľnther
@oliverguenther
Aug 24 2015 11:36
Trying to use a document database for something so extremely relational as project management will yield only pain
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:36
Try that... I'd also check for basic MySQL (mis)configurations, not sure if there are things like connection limits etc.
Very possible that the settings from the default installation are not as optimized.
Vladimir K
@cronacronis
Aug 24 2015 11:38
@oliverguenther agree :( no transaction support neither
@NobodysNightmare did you hear or perform any previous code instrumentation to measure latencies for components?
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:47
We are monitoring the workload on our production instances and for specific performance problems we also analyze the cause and try to improve it.
But I am honest, as far as I know we never had a look at the WP creation performance, at least not to the scope you are aiming for.
That is why I am yet uncertain how far you will get performance-wise. I may have a look at possible bottlenecks later today, if time will let me...
Anyway: If you have a good hint of a bottleneck, I will follow it... My general advise would be to watch out for CPU and IO load during your stress tests. Would be good to know, if either IO or CPU get to their limit... (still we would need to see how to stretch from there on)
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 11:56
@cronacronis I talked to @kgalli he says that you will also need to increase the worker pool of the apache (which is sitting in front of the unicorn). That might explain your problem that 4 unicorn workers are as good as 16...
He will be on his desk soon and might then be able to explain how exactly to re-configure the pool size for the apache.
Vladimir K
@cronacronis
Aug 24 2015 12:13
Great! thx!
kgalli
@kgalli
Aug 24 2015 12:29
Hi @cronacronis I think you want to have a look at the file under /etc/apache2/mods-enabled/mpm_event.conf. Here you can play around with some interesting APACHE settings like MaxRequestWorkers. Good documentation on each parameter can be found here.
Jan Sandbrink
@NobodysNightmare
Aug 24 2015 12:43
Looking at the standard settings we noticed that this will probably not be your problem for 16 unicorn workers...
kgalli
@kgalli
Aug 24 2015 12:46
right. The apache bottleneck will probably appear using more than 50 unicorn processes or so. Do you have some monitoring tool at hand. If not I suggest you give newrelic a chance. Should be easy to setup and it is free (at least for what you need right now).
edit your Gemfile under opt/openproject-ce and then run openproject-ce run bundle install
Vladimir K
@cronacronis
Aug 24 2015 13:08
@kgalli thx. It is set to 150 in fact. I have increased to 1500 now :)
Markus Kahl
@machisuji
Aug 24 2015 21:07
Did it help?