These are chat archives for cboursnell/transfuse

20th
Oct 2015
Will
@willblev
Oct 20 2015 09:37
now I'm trying to get it working on my laptop- Ubuntu 14.04 LTS, i7, 32GB RAM, ruby 1.9.1 ; I'm having general issues with rubygems (don't think it's an issue with transrate/fuse)
/usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:in require': /usr/lib/ruby/gems/1.9.1/gems/transrate-1.0.1/lib/transrate/transrater.rb:23: syntax error, unexpected tLABEL (SyntaxError) def initialize(assembly, reference, threads: 1) ^ /usr/lib/ruby/gems/1.9.1/gems/transrate-1.0.1/lib/transrate/transrater.rb:119: syntax error, unexpected keyword_end, expecting $end end # Transrate ^ from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:54:inrequire'
from /usr/lib/ruby/gems/1.9.1/gems/transrate-1.0.1/lib/transrate.rb:30:in <top (required)>' from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:128:inrequire'
from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:128:in rescue in require' from /usr/local/lib/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:39:inrequire'
from ./transrate:2:in `<main>'
I've got RVM now- do you recommend I try with ruby 2.2.3?
Chris Boursnell
@cboursnell
Oct 20 2015 09:40
Yeah I think it could be something to do with using ruby 1.9...
Chris Boursnell
@cboursnell
Oct 20 2015 09:46
We are going to package transfuse in the same way as transrate to make it a simple and single binary download soon
Will
@willblev
Oct 20 2015 09:51
Splendid. I've been trying to convince my group that a single assembler/assembly based approach is flawed, but until I show them the data from a properly merged assembly they won't change!
Chris Boursnell
@cboursnell
Oct 20 2015 09:53
Cool. I hope it works out. Let me know how it goes :D
Will
@willblev
Oct 20 2015 10:58

/home/will/.rvm/rubies/ruby-2.2.3/bin/ruby -r ./siteconf20151020-5395-vq4asr.rb extconf.rb
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling transrate.c
linking shared-object transrate/transrate.so
/usr/bin/ld: cannot find -lgmp
collect2: error: ld returned 1 exit status
make: * [transrate.so] Error 1

make failed, exit code 2
will@will-Inspiron-5423:~/.rvm/gems/ruby-2.2.3/gems$ ruby -v
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]
will@will-Inspiron-5423:~/.rvm/gems/ruby-2.2.3/gems$ gem env
RubyGems Environment:

  • RUBYGEMS VERSION: 2.4.8
  • RUBY VERSION: 2.2.3 (2015-08-18 patchlevel 173) [x86_64-linux]
  • INSTALLATION DIRECTORY: /home/will/.rvm/gems/ruby-2.2.3
  • RUBY EXECUTABLE: /home/will/.rvm/rubies/ruby-2.2.3/bin/ruby
  • EXECUTABLE DIRECTORY: /home/will/.rvm/gems/ruby-2.2.3/bin
  • SPEC CACHE DIRECTORY: /home/will/.gem/specs
  • SYSTEM CONFIGURATION DIRECTORY: /etc
  • RUBYGEMS PLATFORMS:
    • ruby
    • x86_64-linux
  • GEM PATHS:
  • GEM CONFIGURATION:
    • :update_sources => true
    • :verbose => true
    • :backtrace => false
    • :bulk_threshold => 1000
    • "gem" => "--no-ri --no-rdoc"
  • REMOTE SOURCES:
  • SHELL PATH:
having trouble installing transrate with gem install transrate- will try with binary
Chris Boursnell
@cboursnell
Oct 20 2015 11:02
Transfuse needs the transrate gem to be installed because it needs to call the functions programatically
@Blahah have you seen this error before?
linking shared-object transrate/transrate.so /usr/bin/ld: cannot find -lgmp collect2: error: ld returned 1 exit status
Will
@willblev
Oct 20 2015 11:04
in that case, can you make any sense of the error that was raised above?
Richard Smith-Unna
@blahah
Oct 20 2015 11:38
For some reason your machine doesn't have libgmp installed, which is very unusual
Will
@willblev
Oct 20 2015 12:33
installed libgmp-dev, transrate is working! installing deps (then trying to get transfuse working)
Chris Boursnell
@cboursnell
Oct 20 2015 12:34
:D
Richard Smith-Unna
@blahah
Oct 20 2015 12:37
Awesome
Sorry for the trouble you've had, looks like we need to improve the install for systems like yours
Will
@willblev
Oct 20 2015 12:39
Most people won't be trying to run assemblies/evaluate them/merge them on a laptop running ubuntu ;-)
I'm doing it this way to avoid having to go through our sysadmin
they've got other things to worry about
Will
@willblev
Oct 20 2015 13:37
transfuse appears to be running on my laptop- hopefully it finds/uses the existing files that transrate generated, otherwise I may be waiting for a few hours...
Chris Boursnell
@cboursnell
Oct 20 2015 14:21
If you use the --verbose option it should tell you what it's doing and if it found existing files etc
Will
@willblev
Oct 20 2015 14:27
I can see that it's not finding/using the preexisting files, but that's probably because they were from transrate 1.0.0.beta3 (and I'm guessing the naming/file structure has changed)
Will
@willblev
Oct 20 2015 14:57
it's only 20% slower on my laptop as compared to the cluster X-D
perhaps in future versions transfuse could run transrate on each assembly simultaneously (parallel) instead going through 1 by 1?
Chris Boursnell
@cboursnell
Oct 20 2015 15:01
well if you set the threads to be equal to the number of cpus you have in the computer it will be parallel
Will
@willblev
Oct 20 2015 15:08
using quad-core computer, and set threads to 3 (keeping one spare to allow me to perform other low-memory tasks)
as I saw the files from transrate for each assembly appearing without overlap between the modification times (and in the same order that they appeared in the -a argument, as opposed to the size or complexity of each assembly) i assumed that transfuse wasn't evaluating multiple assemblies at once
Chris Boursnell
@cboursnell
Oct 20 2015 15:15
Yeah it runs transrate on one after the other, but it runs transrate with 3 (in your case) threads.
Will
@willblev
Oct 20 2015 15:15
wait I think I understand now... you mean that it splits the job of running transrate on EACH assembly across multiple threads, not that it tries to run transrate on separate assemblies at once
Chris Boursnell
@cboursnell
Oct 20 2015 15:19
If you give the option to transfuse --assemblies A,B,C --threads 8 it will run transrate on assembly A with 8 threads, then next it will run transrate on B with 8 threads...
Will
@willblev
Oct 20 2015 15:22
got it. maybe it's running fast-ish because i have a SSD? transrate generates ~40G so the read/write time is probably a significant chunk of the 2h it takes me to run it
Chris Boursnell
@cboursnell
Oct 20 2015 15:35
There are some parts of transrate that are not multithreaded and are reading files from the disk, but they're not IO bound. Running with 3 threads you wouldn't be IO bound on a hard disk either.
Will
@willblev
Oct 20 2015 15:50
since i'm probably going to run transfuse on 20 other samples with 25M paired-end reads each, do you have a suggestion on how many cores/how much memory I should dedicate (is there a performance dropoff at some point)? my options are nodes with 12 cores/32GB RAM, 24c/96GB, or 32c/128GB
Chris Boursnell
@cboursnell
Oct 20 2015 15:55
I usually run it with between 24 and 48 threads. Running vsearch as part of transfuse takes the longest time. For merging 3 assemblies it usually takes about 48 hours. The speed of vsearch will be governed by the number of contigs in the fasta files. The speed of the transrate step with be based on the number of reads. The majority of the cpu time will scale pretty linearly with number of threads
Will
@willblev
Oct 20 2015 15:59
I'm very glad you told me that. I was waiting for vsearch to finish on my laptop so I could head home for dinner, and I think I would have starved to death
Chris Boursnell
@cboursnell
Oct 20 2015 16:02
We would like to use a faster program for the clustering. However this is the only one we have found that gives all the information we need and clusters transcripts correctly.
Rob Patro
@rob-p
Oct 20 2015 21:42
@cboursnell — we're working on a faster clustering tool based on the quasi-mapping concept. Do you have some suggestions on datasets that would be good to test on?
Chris Boursnell
@cboursnell
Oct 20 2015 22:52
I'm not sure what would be a good test dataset. You want to be able to know what is the 'correct' clustering?
I have sets of fasta files that I ran through transfuse while I was developing that. I can send you those?
That's very cool :D I'll look forward to playing with it when you have a beta :P