These are chat archives for collectiveaccess/support

6th
Apr 2017
Kehan Harman
@kehh
Apr 06 2017 02:22
@murchmurch we've encountered concurrency problems when running caUtils process-task-queue. If your queue is taking long I'd suggest making sure you're using the gmagick php extension rather than the other extensions - it is at least one if not two orders of magnitude faster than the others (don't ask me how).
Kehan Harman
@kehh
Apr 06 2017 02:27
For your first question, you can change the number of thumbnails by editing/overriding app/conf/media_procesing.conf - you'll have to update at least theVERSIONS key of each media type within each media table, but might also want to update the MEDIA_TRANSFORMATION_RULES for each media table.
CollectiveAccess
@collectiveaccess
Apr 06 2017 03:28
@kehh What problems have you seen running multiple instances of process-task-queue? We run it often in parallel without any issues
Regarding gmagick, it's faster because it's using GraphicsMagick which is... well it's just faster than ImageMagick. The GraphicsMagick developers prioritize speed and memory footprint over # of formats
Kehan Harman
@kehh
Apr 06 2017 03:31
@collectiveaccess I think it was more that we had a cron job to process the queue, and because there were loads of media to import we ended up with too many cron jobs running at once and unexpected server load.
So we implemented an external locking system for the queue. But maybe our cron jobs were too close (and I'm pretty sure this was because we weren't using GraphicsMagick yet).
CollectiveAccess
@collectiveaccess
Apr 06 2017 03:33
There is a locking system for the queue, but it relies upon the PHP POSIX extension
At the time it was written it was the only way to figure out what was running. There's probably a better way to do it with PHP 7.x; maybe we'll get to rewriting that before 2020
Kehan Harman
@kehh
Apr 06 2017 03:35
I'd definitely seen that in the search indexing queue, but wasn't sure if it existed in the task queue.
CollectiveAccess
@collectiveaccess
Apr 06 2017 03:35
If you don't have POSIX installed it false back to a very unreliable guesswork system
It works well with POSIX; I know because we have some large installs that are cranking 4 queues at the same time 24 hours a day
and they never scream at me (for that at least)
(this is what I'm talking about: http://php.net/manual/en/book.posix.php)
Kehan Harman
@kehh
Apr 06 2017 03:44
posix seems to be enabled by default. It's probably due to some dodgy plugins that we wrote that hook into the queue as well (https://github.com/wamuseum/providence/tree/master-fix/app/plugins/wamMaterialisedTaxonomy)
CollectiveAccess
@collectiveaccess
Apr 06 2017 04:52
I've noticed that not all distributions enable posix by default... but who knows
murchmurch
@murchmurch
Apr 06 2017 06:34
@kehh Thanx for the anwsers!! Im running on a Windows machine an i have tried to install the gmagic php extension but PHP just wont start when i enable it. Now using gm.exe for the file conversion (still quite slow). I anyone has any input on how to get gmagic php extension to work in a windows enviroment pelase share ;)
Kehan Harman
@kehh
Apr 06 2017 07:48
+1 for the v1.7 release candidate @collectiveaccess @juliaweist @pinkfishies
CollectiveAccess
@collectiveaccess
Apr 06 2017 11:14
@murchmurch I don't think I have ever actually seen gmagick run on Windows
gm.exe is going to be slow as it has to spawn a command line process for each derivative, and quite a few to tile the image for zoom
Building gmagick from source is probably the only way to get it to work. I might give that a try. What are the parameters for your system? 64-bit or 32-bit? Which VC runtime? (Eg. VC10, VC13 etc.)?
murchmurch
@murchmurch
Apr 06 2017 12:39
@collectiveaccess Im running a 32-bit apache/php 5.6 windows 2012 server enviroment. I can install any Visual C ++ runtime lib that you recomend. No other dependencies on the server.
CollectiveAccess
@collectiveaccess
Apr 06 2017 13:44
@murchmurch What matters is what VC runtime the PHP you're running is compiled under
yelto
@yelto
Apr 06 2017 16:20
Windows install is finicky, but we also managed to complete an install running gm.exe. It is really slow compared to a php gm extension on linux, but it does work. The whole thing runs on php-7.0 VC14 x64. We'd be interested to help figure out the php extension install, it should give much better performance. @collectiveaccess Congrats and thanks for 1.7!
murchmurch
@murchmurch
Apr 06 2017 17:50

@collectiveaccess PHP Version 5.6.30

System Windows NT WIN-7JVQRHLPGTT 6.2 build 9200 (Windows Server 2012 Standard Edition) i586
Build Date Jan 18 2017 19:42:12
Compiler MSVC11 (Visual C++ 2012)
Architecture x86

VC11 it is then..
murchmurch
@murchmurch
Apr 06 2017 20:30
I found something a little odd. I can start 4 CMD windows that each run the "php caUtils process-task-queue" command. But when i open a 5 window and try to run the same command i get Processing compleate. Is there a limit of 4 pararell prcesses that can run or can i increse this? I have 100.000 images to process thats why i wnat to be able to do it in pararell.
this is how it looks:
blob
murchmurch
@murchmurch
Apr 06 2017 20:37
I also found that the "php caUtils process-task-queue" ends (in all 4 cmd windows) and when I re-run the same command it starts again. Clearly it didn’t process all the pictures. Is there some way to control this behaviour? The goal would be to have the process run for as long there are files to process. Sorry for spamming so many questions…