camillescott/dammit

just annotate it, dammit!

Camille Scott

@camillescott

i'm gonna guess that its a problem with gnu parallel

can you try running which parallel?

ulrichkudahl

@ulrichkudahl

"/usr/local/bin/parallel"

Camille Scott

@camillescott

gotcha

in that case

do tail -n 500 ~/.dammit/log/dammit-all.log and post the result here :)

ulrichkudahl

@ulrichkudahl

04-22 22:52:30 dammit.annotate:write:85 [INFO]

[x] transcriptome_stats:test2.fasta

04-22 22:52:30 dammit.annotate:write:85 [INFO]

[x] busco:test2.fasta-metazoa

04-22 22:52:30 dammit.annotate:write:85 [INFO]

[x] TransDecoder.LongOrfs:test2.fasta

04-22 22:52:30 dammit.annotate:write:85 [INFO]

[ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

04-22 22:52:30 dammit.annotate:write:85 [ERROR]

Some tasks failed!

04-22 22:52:30 dammit.annotate:write:85 [ERROR]

TaskFailed - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm

04-22 22:52:30 dammit.annotate:write:85 [ERROR]
Command failed: 'S=stat -c "%s" test2.fasta.transdecoder_dir/longest_orfs.pep; B=expr $S / 1; cat test2.fasta.transdecoder_dir/longest_orfs.pep | /usr/local/bin/parallel --block $B --pipe --recstart ">" --gnu -j 1 /usr/local/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o test2.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /Users/Ulrich/.dammit/databases/Pfam-A.hmm /dev/stdin > test2.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 20

04-22 22:52:30 dammit.annotate:write:85 [DEBUG]

04-22 22:52:30 dammit.annotate:write:85 [DEBUG]
stat: illegal option -- c
usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
expr: syntax error

Camille Scott

@camillescott

ahhh

looks like this is one of those "OSX install *nix utilities with the same name/purpose and slightly different options" problems

ulrichkudahl

@ulrichkudahl

Yes, the stat command doesn't respond to the -c option

Camille Scott

@camillescott

alrighty, here's what i'll do

i'll push a fix to github in a bit

and you can pip install using the dev instructions

ulrichkudahl

@ulrichkudahl

pip install git+https://github.com/camillescott/dammit.git

right?

Camille Scott

@camillescott

yup!

i'll let you know when i've got a fix

sorry for this!

ulrichkudahl

@ulrichkudahl

Don't apologise :) I am just amazed that you respond to my query

Camille Scott

@camillescott

haha

i try to keep an eye on everything, but some problems are easier to diagnose than others ;)

ulrichkudahl

@ulrichkudahl

Hi Camille. Since we spoke last I rented a virtual unix machine and setup dammit there and have successfully run dammit on a set of differentially expressed genes from an RNA-seq study. As an overwhelming about of results files are produced, I am a bit unsure how to approach the analysis side of things. How and with which output files would you recommend starting?

Matt MacManes

@macmanes

Fwiw I basically only use the final fasta and gff3 files which are basically a sum of all the annotation information.

ulrichkudahl

@ulrichkudahl

Thanks again for the quick reply. I suppose I am still a bit unsure about how to really use the GFF3 files, but I will give it a go

On another note, I tried do to a full install of the databases, but I keep getting an out of memory error. How much memory do you recon I would need do to a full database installation?

Matt MacManes

@macmanes

Memory or hard drive space?

ulrichkudahl

@ulrichkudahl

Memory, but I am also curious to know how much space is required

*hard drive space is required

Matt MacManes

@macmanes

about use of GFF3. I also use in IGV for viz purposes. Honestly I’ve neved kept track of RAM usage for making te databases. My dammit database folder is 59G

Camille Scott

@camillescott

the databases can be quite memory intensive to generate; expect ~16gb for the orthodb one

not sure what is necessary for uniref90, though i can try to find out for you

Camille Scott

@camillescott

currently dammit doesn't have an option to custom tune parameters

however, if you're willing to get a little dirty, you can tune them yourself

if you can find your site-packages directory, you can edit the file. something like /path/to/python/lib/python2.7/site-packages/dammit

there's a file in there called .config.json

one group is called "last", and has an entry for "lastdb"

you can add extra arguments to its params entry, and they will be used in the run

there is a bunch of info about tuning here: http://last.cbrc.jp/doc/last-tuning.html

the only extra param included by default is -w3

this reduces memory usage

you can up that to -w5 or more

there are tradeoffs, explained in the last docs i linked

hope that helps @ulrichkudahl :)

future version will probably allow for more tuning (ie passing in your own config file)

Matthew J. Oldach

@moldach

Hello Camille,

I hope this message finds you well.

I'm running into an issue with dammit annotate and hoping you can help.

I ran dammit annotate /home/ubuntu/V10.Trinity.fasta --busco-group eukaryota --database-dir /home/ubuntu/dammit/ --full and got the following error:

--- Checking PATH for dependencies

          [x] HMMER

          [x] Infernal

          [x] BLAST+

          [x] BUSCO

          [x] TransDecoder

          [x] LAST

          [x] crb-blast


--- Dependency results

          All dependencies satisfied!


--- Checking for database prep (dir: /home/ubuntu/dammit)

          [x] download_and_gunzip:Pfam-A.hmm

          [x] hmmpress:Pfam-A.hmm

          [x] download_and_gunzip:Rfam.cm

          [x] cmpress:Rfam.cm

          [x] download_and_gunzip:aa_seq_euk.fasta

          [x] lastdb:aa_seq_euk.fasta.db

          [x] download_and_gunzip:ODB8_EukOGs_genes_ALL_levels.txt

          [x] download_and_untar:dammit-eukaryota

          [x] download_and_gunzip:uniref90.fasta

          [x] lastdb:uniref90.fasta.db


--- Database results

          All databases prepared!


--- Running annotate!

          Transcriptome file: V10.Trinity.fasta

          Output directory: /home/ubuntu/dammit/V10.Trinity.fasta.dammit

          [x] V10.Trinity.fasta

          [x] transcriptome_stats:V10.Trinity.fasta

          [x] busco:V10.Trinity.fasta-eukaryota

          [x] TransDecoder.LongOrfs:V10.Trinity.fasta

          [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

Some tasks failed![dammit.annotate:ERROR]
TaskError - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm[dammit.annotate:ERROR]
Command error: 'cat V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep | None --round-robin --pipe -L 2 -N 10000 --gnu -j 1 -a V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep /usr/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /home/ubuntu/dammit/Pfam-A.hmm /dev/stdin > V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 127
[dammit.annotate:ERROR]

As per your earlier comments with @ulrichkudahl I tried which parallel and found that it was not installed. After sudo apt-get install parallel I re-ran the dammit annotate command and a new error was returned.

...
[ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

Some tasks failed![dammit.annotate:ERROR]
TaskError - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm[dammit.annotate:ERROR]
Command error: 'cat V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep | /usr/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 36 -a V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep /usr/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /home/ubuntu/dammit/Pfam-A.hmm /dev/stdin > V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 255
[dammit.annotate:ERROR]

Your help is greatly appreciated.
Thanks,
Matt

Matt MacManes

@macmanes

Hey @moldach, which version of gnu parallel do you have installed? I remember that dammit requires a newer version than I initially had. I used Linuxbrew to install.

Matthew J. Oldach

@moldach

Thanks for the tip @macmanes brew install parallel worked!

b4ph0m3t

@b4ph0m3t

hello I am trying to run dammit over the provided sample data in a CentOS machine.
It runs until


          [ ] lastal:cdna_nointrons_utrs.fa.x.orthodb.maf

Some tasks failed![dammit.annotate:ERROR]
TaskFailed - taskid:lastal:cdna_nointrons_utrs.fa.x.orthodb.maf[dammit.annotate:ERROR]
Command failed: 'cat cdna_nointrons_utrs.fa | /usr/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 1 -a cdna_nointrons_utrs.fa "/home/labgen/Software/last-658/src/lastal -F15 -D100000.0 /home/labgen/.dammit/databases/aa_seq_euk.fasta.db" > cdna_nointrons_utrs.fa.x.orthodb.maf' returned 1
[dammit.annotate:ERROR]

happens.
Is it a dependency issue?
Thanks

b4ph0m3t

@b4ph0m3t

solved the error by deleting .prj file from /databases/ folder and re-running

dammit databases --install

there was some strange issue with lastal reading .prj file (some lines were missing)

Matthew J. Oldach

@moldach

It appear like dammit is working, although maybe not a 100% since not all of the boxes were checked:

ubuntu@ip-172-31-2-75:~/dammit$ dammit annotate /home/ubuntu/good.V10.Trinity.fasta --busco-group metazoa --database-dir /home/ubuntu/dammit/ --full
========================================
dammit! a tool for easy de novo transcriptome annotation
Camille Scott 2015
========================================

~~~~~~~~~~~~~~~~~~~~~~~~

submodule: annotate


--- Checking PATH for dependencies

          [x] HMMER

          [x] Infernal

          [x] BLAST+

          [x] BUSCO

          [x] TransDecoder

          [x] LAST

          [x] crb-blast


--- Dependency results

          All dependencies satisfied!


--- Checking for database prep (dir: /home/ubuntu/dammit)

          [x] download_and_gunzip:Pfam-A.hmm

          [x] hmmpress:Pfam-A.hmm

          [x] download_and_gunzip:Rfam.cm

          [x] cmpress:Rfam.cm

          [x] download_and_gunzip:aa_seq_euk.fasta

          [x] lastdb:aa_seq_euk.fasta.db

          [x] download_and_gunzip:ODB8_EukOGs_genes_ALL_levels.txt

          [x] download_and_untar:dammit-metazoa

          [x] download_and_gunzip:uniref90.fasta

          [x] lastdb:uniref90.fasta.db


--- Database results

          All databases prepared!


--- Running annotate!

          Transcriptome file: good.V10.Trinity.fasta

          Output directory: /home/ubuntu/dammit/good.V10.Trinity.fasta.dammit

          [x] good.V10.Trinity.fasta

          [x] transcriptome_stats:good.V10.Trinity.fasta

          [x] busco:good.V10.Trinity.fasta-metazoa

          [x] TransDecoder.LongOrfs:good.V10.Trinity.fasta

          [x] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

          [x] remap_hmmer:longest_orfs.pep.pfam.tbl

          [ ] TransDecoder.Predict:good.V10.Trinity.fasta

          [x] cmscan:good.V10.Trinity.fasta.x.Rfam.cm

          [x] lastal:good.V10.Trinity.fasta.x.orthodb.maf

          [x] lastal:good.V10.Trinity.fasta.x.uniref.maf

          [x] maf_best_hits:good.V10.Trinity.fasta.x.orthodb.maf-
          good.V10.Trinity.fasta.x.orthodb.maf.best.csv

          [x] maf-gff3:good.V10.Trinity.fasta.x.orthodb.maf.gff3

          [x] maf_best_hits:good.V10.Trinity.fasta.x.uniref.maf-
          good.V10.Trinity.fasta.x.uniref.maf.best.csv

          [x] maf-gff3:good.V10.Trinity.fasta.x.uniref.maf.best.csv.gff3

          [x] hmmscan-gff3:good.V10.Trinity.fasta.pfam.csv.gff3

          [x] cmscan-gff3:good.V10.Trinity.fasta.rfam.tbl.gff3

          [ ] gff3-merge:good.V10.Trinity.fasta.dammit.gff3

          [x] fasta-annotate:good.V10.Trinity.fasta.dammit.fasta

Does this mean TransDecoder.Predict and gff3-merge failed to run?
Any suggestions for how to force them to complete if so? I tried running the command a number of times to no avail

Thank you

Where communities thrive

People

Repo info

Activity

[x] transcriptome_stats:test2.fasta

[x] busco:test2.fasta-metazoa

[x] TransDecoder.LongOrfs:test2.fasta

[ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

Some tasks failed!

TaskFailed - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm