These are chat archives for dereneaton/ipyrad

17th
Aug 2016
Cycadales
@Cycadales
Aug 17 2016 18:49
@dereneaton @isaacovercast Sorry another quick question with regards to on HPC clusters. We are testing a system/systems with 4 x 8 core + 32gb ram each with ipyrad and have ran into some problems. The code we were trying was as follows ipyrad -p params-CaXm.txt -s 12 -c 32 -r --MPI. However, when we ran this script it seemed to just load everything to one system and not use any resources on the other systems connected to the cluster.
Isaac Overcast
@isaacovercast
Aug 17 2016 18:51
Hey James, Is this what you were saying earlier was only running on one compute node?
Cycadales
@Cycadales
Aug 17 2016 18:53
@isaacovercast but @duncanreddish_twitter could give you more info than me on this as I am only trying to remember what he typed!
@isaacovercast we did also try and specify the nodes using a pre MPI command and provide the number cores and then run ipyrad and it just went crazy!
Deren Eaton
@dereneaton
Aug 17 2016 18:57
@Cycadales to be clear, the ipyrad command is meant to be used on its own, i.e., not like: mpirun -n 32 ipyrad ...
Cycadales
@Cycadales
Aug 17 2016 19:00
well we tried it both ways mpirun -n 32 ipyrad ... like you said did not work at all (kind of went crazy). So if we were to run ipyrad -p params-CaXm.txt -s 12 -c 32 -r --MPI this would be the correct way yes?
Deren Eaton
@dereneaton
Aug 17 2016 19:00
yeah, otherwise it will go pretty crazy, ha.
When you run it the second way do you see only 8 cores running?
Cycadales
@Cycadales
Aug 17 2016 19:04
So it must be something on our end. Because when we tried ipyrad -p params-CaXm.txt -s 12 -c 32 -r --MPI just loads everything to one system and does not use the others at all. It seems to try add the load of 32 cores onto one 8 core system.
Deren Eaton
@dereneaton
Aug 17 2016 19:05
are you using a 'qsub' HPC setup?
Cycadales
@Cycadales
Aug 17 2016 19:07
I am not sure to be honest as I did not set up the system. This is something @duncanreddish_twitter would know and he might be able to ask a few questions tomorrow. is the 'qsub' the one you would recommend?
Deren Eaton
@dereneaton
Aug 17 2016 19:09
are you using a personal cluster or your University's cluster? qsub is a command to submit jobs to a cluster on many University clusters.
Cycadales
@Cycadales
Aug 17 2016 19:10
this would be a personal cluster setup by the IT manager at the Royal Botanic Garden Edinburgh.
Deren Eaton
@dereneaton
Aug 17 2016 19:12
I see. They're all usually pretty similar, so it could be a problem with on our end. We're still optimizing parallelization across many nodes. I've done testing recently on 40 cores across 5 nodes, but I'll check that it's still working on the latest version.
Cycadales
@Cycadales
Aug 17 2016 19:16
Sorry I don't know more about the RBGE cluster. But if you could rule out the ipyrad end that would be a huge help for us. I have a number of large dataset to run and the University of Edinburgh is kind of ruled out due to cost, as we spent so much on sequencing. So this is why we are trying at the RBGE. Plus this could help other too who are wanting to do RADseq.
Deren Eaton
@dereneaton
Aug 17 2016 19:17
of course. I'm doing some testing now.
In the mean time, you can probably still assemble a reasonable sized data set with just 8 cores pretty quickly.
Or you might as well start, that is, since you could stop the job later and restart it from wherever it is at with more cores if we get it working.
Cycadales
@Cycadales
Aug 17 2016 19:23
Well I can wait a little bit as I am running a large job in Sydney (one with the error in step 4). With the data sets that I need to run here I have already done the testing (usually just one population). But the other are 82 and 106 samples 150 PE read data so are quite large. It did seem to take forever on my Apple MBP (4 cores, 8 threads) just to run 10 samples. But I suppose it is a mobile processor.
Deren Eaton
@dereneaton
Aug 17 2016 22:31
@Cycadales It appears to be working just fine. Do you know how you were confirming that it was only running on one host? If you use the command top on a single computer it will only show the processes running on that host, but not on remote hosts. But if you run with -c 16 versus -c 4 you should see it run 4X faster. I made some changes to make it apparent that a multi-host connection is established. I'll add this to the next version update.
[de243@compute-21-11 ~]$ ipyrad -p params-pedictest.txt -s 2 -f -c 16 --MPI


 -------------------------------------------------------------
  ipyrad [v.0.3.29]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  loading Assembly: pedictest
  from saved path: ~/pedictest/pedictest.json
  Establishing MPI connection to remote hosts
  Host parallel setup: [12 cores] on compute-21-5.local
  Host parallel setup: [4 cores] on compute-21-11.local

  Step2: Filtering reads 
  [####################] 100%  processing reads      | 0:05:43
Cycadales
@Cycadales
Aug 17 2016 22:35
@dereneaton Thank you very much for this. I will check this in the morning.