These are chat archives for dereneaton/ipyrad

4th
Mar 2018
tim-oconnor
@tim-oconnor
Mar 04 2018 05:32 UTC

@isaacovercast I wrote a while ago about problems with denovo-reference assemblies and you asked for an excerpt of the log file. I was able to dredge up the log from the relevant runs. Here's a few examples:

 -------------------------------------------------------------
  ipyrad [v.0.7.22]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  Begin run: 2018-02-19 10:29
  Using args {'preview': False, 'force': True, 'threads': 2, 'results': False, 'quiet': False, 'merge': None, 'ipcluster': None, 'cores': 22, 'params': 'params-s345-6x-c90.txt', 'branch': None, 'steps'
: '345', 'debug': False, 'new': None, 'download': None, 'MPI': False}
  Platform info: ('Linux', 'n0292.savio2', '3.10.0-693.11.6.el7.x86_64', '#1 SMP Wed Jan 3 18:09:42 CST 2018', 'x86_64')2018-02-19 10:54:52,815     pid=16321     [assembly.py]    ERROR     OSError([
Errno 2] No such file or directory: '/global/scratch/toconnor/ipyrad/larrea/s12_edits/AZAO_705_4x-refmap_derep.fastq')
2018-02-19 11:12:15,699     pid=16761     [assembly.py]    ERROR     IOError([Errno 2] No such file or directory: '/global/scratch/toconnor/ipyrad/larrea/s345-6x-c85_clust_0.85/CAI_514_6x.utemp.sort
')

 -------------------------------------------------------------
  ipyrad [v.0.7.22]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  Begin run: 2018-02-19 11:12
  Using args {'preview': False, 'force': True, 'threads': 2, 'results': False, 'quiet': False, 'merge': None, 'ipcluster': None, 'cores': 22, 'params': 'params-s345-6x-c90.txt', 'branch': None, 'steps'
: '345', 'debug': False, 'new': None, 'download': None, 'MPI': False}
  Platform info: ('Linux', 'n0058.savio2', '3.10.0-693.11.6.el7.x86_64', '#1 SMP Wed Jan 3 18:09:42 CST 2018', 'x86_64')2018-02-19 11:37:31,655     pid=16702     [assembly.py]    ERROR     IOError([
Errno 2] No such file or directory: '/global/scratch/toconnor/ipyrad/larrea/s12_edits/CAI_514_6x-refmap_derep.fastq')

 -------------------------------------------------------------
  ipyrad [v.0.7.22]
  Interactive assembly and analysis of RAD-seq data
 -------------------------------------------------------------
  Begin run: 2018-02-19 11:37
  Using args {'preview': False, 'force': True, 'threads': 2, 'results': False, 'quiet': False, 'merge': None, 'ipcluster': None, 'cores': 22, 'params': 'params-s345-6x-c93.txt', 'branch': None, 'steps'
: '345', 'debug': False, 'new': None, 'download': None, 'MPI': False}
  Platform info: ('Linux', 'n0292.savio2', '3.10.0-693.11.6.el7.x86_64', '#1 SMP Wed Jan 3 18:09:42 CST 2018', 'x86_64')2018-02-19 12:20:20,606     pid=17936     [assembly.py]    ERROR     IOError([
Errno 2] No such file or directory: '/global/scratch/toconnor/ipyrad/larrea/s12_edits/T5Q4_2038_6x-refmap_derep.fastq')

Not sure if that is enlightening. Sorry it took so long.

Isaac Overcast
@isaacovercast
Mar 04 2018 17:26 UTC
@tim-oconnor Well the only thing I can think is that you are running out of disk space, either raw disk or you're hitting a quota if this is on an HPC system. This would explain both the error (file not found) and the seeming non-determinism (different outcome every time). It would also potentially explain why denovo-reference sometimes fails, but not denovo, as the denovo-reference assembly step still generates all the intermediate reference assembly files which can be quite large. Can you ensure there is enough space on /global/scratch and that you aren't encountering a quota limit?
@JStarrett The most appropriate params values are going to be unique to each dataset, so you should feel free to explore different values. The cannonical target dataset (Pedicularis) is 100bp SE, but I can't say the default params are exactly specific to this.
@gissella_pineda_twitter Can you stop step 5 and restart it with the -d flag? this will generate debug output in the ipyrad_log.txt file, which should help us figure out what's going on.
tim-oconnor
@tim-oconnor
Mar 04 2018 19:04 UTC
@isaacovercast Thanks for clearing that up! In principal there is no quota on /global/scratch, but the disk space is so full that the administrators are instituting a data purge. I must have bumped up against the system's disk space.