These are chat archives for dereneaton/ipyrad

10th
Jan 2018
markusruhsam
@markusruhsam
Jan 10 2018 08:23 UTC

@dereneaton I am having problems 'with an 'unexpected error' during the database indels process at step 6. Here's what I get


ipyrad [v.0.7.17]

Interactive assembly and analysis of RAD-seq data

loading Assembly: Araucaria_min2
from saved path: /mirror/5tb/markus/Araucaria/Araucaria_min2.json
establishing parallel connection:
host compute node: [10 cores] on hpc5
host compute node: [10 cores] on hpc6
host compute node: [10 cores] on hpc1
host compute node: [10 cores] on hpc0
host compute node: [10 cores] on hpc3
host compute node: [10 cores] on hpc2

Step 6: Clustering at 0.85 similarity across 41 samples

[####################] 100% concat/shuffle input | 0:00:40
[####################] 100% clustering across | 4:44:32
[####################] 100% building clusters | 0:00:53
[####################] 100% aligning clusters | 0:01:47
[####################] 100% database indels | 0:05:02
[ ] 0% indexing clusters | 0:00:04
Encountered an unexpected error (see ./ipyrad_log.txt)
Error message is below -------------------------------
MemoryError()
warning: error during shutdown:
[Errno 3] No such process

Summary stats of Assembly Araucaria_min2

                  state  reads_raw  reads_passed_filter  clusters_total  \

870749PairedTrim 5 5341746 5341030 3993488
NC01_029PairedTrim 5 5545542 5544442 3804505
NC01_332PairedTrim 5 3885478 3884952 3029628
NC01_670PairedTrim 5 3916215 3915731 3004918
NC01_687PairedTrim 5 7641144 7639852 3889811
NC01_705PairedTrim 5 5338608 5337902 3523462
NC01_737PairedTrim 5 3654841 3654307 2751929
...........

The last entry in the ipyrad_log shows this

Begin run: 2018-01-09 07:16
Using args {'preview': False, 'force': True, 'threads': 2, 'results': True, 'quiet': False, 'merge': None, 'ipcluster': None, 'cores': 70, 'params': 'params-Araucaria_min2.txt', 'branch': None, 'steps': '67',
'debug': False, 'new': None, 'MPI': True}
Platform info: ('Linux', 'hpc6', '4.4.0-21-generic', '#37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016', 'x86_64')2018-01-09 12:09:18,846 pid=83810 [assembly.py] ERROR MemoryError()
2018-01-09 12:09:19,929 pid=83810 [assembly.py] ERROR shutdown warning: [Errno 3] No such process

Any idea what the problem is?
Thanks very much
Markus

alexjvr1
@alexjvr1
Jan 10 2018 16:18 UTC
@dereneaton Error on Step 5 (ipyrad v.0.7.20)
I'm running PE ddRAD data through jupyter notebook on a remote cluster
Steps 1-4 work without an error. But on step 5 I get the following error message:

Exception: one or more exceptions from call to method: consensus
[8:apply]: ValueError: could not broadcast input array from shape (308,4) into shape (300,4)
[5:apply]: ValueError: could not broadcast input array from shape (302,4) into shape (300,4)
[6:apply]: ValueError: could not broadcast input array from shape (301,4) into shape (300,4)
[3:apply]: ValueError: could not broadcast input array from shape (303,4) into shape (300,4)
.... 29 more exceptions ...
[8:apply]:
ValueErrorTraceback (most recent call last)<string> in <module>()
/home/alexjvr/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/consens_se.pyc in consensus(args)
246 [numpy.sum(arrayed == i, axis=0) for i in list("CATG")],
247 dtype='uint32').T
--> 248 catarr[counters["nconsens"]][:catg.shape[0]] = catg
249 ## store data for tmpchunk
250 storeseq[counters["name"]] = consens
ValueError: could not broadcast input array from shape (308,4) into shape (300,4)

[5:apply]:
ValueErrorTraceback (most recent call last)<string> in <module>()
/home/alexjvr/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/consens_se.pyc in consensus(args)
246 [numpy.sum(arrayed == i, axis=0) for i in list("CATG")],
247 dtype='uint32').T
--> 248 catarr[counters["nconsens"]][:catg.shape[0]] = catg
249 ## store data for tmpchunk
250 storeseq[counters["name"]] = consens
ValueError: could not broadcast input array from shape (302,4) into shape (300,4)

[6:apply]:
ValueErrorTraceback (most recent call last)<string> in <module>()
/home/alexjvr/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/consens_se.pyc in consensus(args)
246 [numpy.sum(arrayed == i, axis=0) for i in list("CATG")],
247 dtype='uint32').T
--> 248 catarr[counters["nconsens"]][:catg.shape[0]] = catg
249 ## store data for tmpchunk
250 storeseq[counters["name"]] = consens
ValueError: could not broadcast input array from shape (301,4) into shape (300,4)

[3:apply]:
ValueErrorTraceback (most recent call last)<string> in <module>()
/home/alexjvr/miniconda2/lib/python2.7/site-packages/ipyrad/assemble/consens_se.pyc in consensus(args)
246 [numpy.sum(arrayed == i, axis=0) for i in list("CATG")],
247 dtype='uint32').T
--> 248 catarr[counters["nconsens"]][:catg.shape[0]] = catg
249 ## store data for tmpchunk
250 storeseq[counters["name"]] = consens
ValueError: could not broadcast input array from shape (303,4) into shape (300,4)

... 29 more exceptions ...

Any idea what the problem could be? Thanks for the help.
Alex