Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    ulrichkudahl
    @ulrichkudahl
    "/usr/local/bin/parallel"
    Camille Scott
    @camillescott
    gotcha
    in that case
    do tail -n 500 ~/.dammit/log/dammit-all.log and post the result here :)
    ulrichkudahl
    @ulrichkudahl

    04-22 22:52:30 dammit.annotate:write:85 [INFO]

    [x] transcriptome_stats:test2.fasta

    04-22 22:52:30 dammit.annotate:write:85 [INFO]

    [x] busco:test2.fasta-metazoa

    04-22 22:52:30 dammit.annotate:write:85 [INFO]

    [x] TransDecoder.LongOrfs:test2.fasta

    04-22 22:52:30 dammit.annotate:write:85 [INFO]

    [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

    04-22 22:52:30 dammit.annotate:write:85 [ERROR]

    Some tasks failed!

    04-22 22:52:30 dammit.annotate:write:85 [ERROR]

    TaskFailed - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm

    04-22 22:52:30 dammit.annotate:write:85 [ERROR]
    Command failed: 'S=stat -c "%s" test2.fasta.transdecoder_dir/longest_orfs.pep; B=expr $S / 1; cat test2.fasta.transdecoder_dir/longest_orfs.pep | /usr/local/bin/parallel --block $B --pipe --recstart ">" --gnu -j 1 /usr/local/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o test2.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /Users/Ulrich/.dammit/databases/Pfam-A.hmm /dev/stdin > test2.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 20


    04-22 22:52:30 dammit.annotate:write:85 [DEBUG]


    04-22 22:52:30 dammit.annotate:write:85 [DEBUG]
    stat: illegal option -- c
    usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
    expr: syntax error


    Camille Scott
    @camillescott
    ahhh
    looks like this is one of those "OSX install *nix utilities with the same name/purpose and slightly different options" problems
    ulrichkudahl
    @ulrichkudahl
    Yes, the stat command doesn't respond to the -c option
    Camille Scott
    @camillescott
    alrighty, here's what i'll do
    i'll push a fix to github in a bit
    and you can pip install using the dev instructions
    ulrichkudahl
    @ulrichkudahl
    right?
    Camille Scott
    @camillescott
    yup!
    i'll let you know when i've got a fix
    sorry for this!
    ulrichkudahl
    @ulrichkudahl
    Don't apologise :) I am just amazed that you respond to my query
    Camille Scott
    @camillescott
    haha
    i try to keep an eye on everything, but some problems are easier to diagnose than others ;)
    ulrichkudahl
    @ulrichkudahl
    Hi Camille. Since we spoke last I rented a virtual unix machine and setup dammit there and have successfully run dammit on a set of differentially expressed genes from an RNA-seq study. As an overwhelming about of results files are produced, I am a bit unsure how to approach the analysis side of things. How and with which output files would you recommend starting?
    Matt MacManes
    @macmanes
    Fwiw I basically only use the final fasta and gff3 files which are basically a sum of all the annotation information.
    ulrichkudahl
    @ulrichkudahl
    Thanks again for the quick reply. I suppose I am still a bit unsure about how to really use the GFF3 files, but I will give it a go
    On another note, I tried do to a full install of the databases, but I keep getting an out of memory error. How much memory do you recon I would need do to a full database installation?
    Matt MacManes
    @macmanes
    Memory or hard drive space?
    ulrichkudahl
    @ulrichkudahl
    Memory, but I am also curious to know how much space is required
    *hard drive space is required
    Matt MacManes
    @macmanes
    about use of GFF3. I also use in IGV for viz purposes. Honestly I’ve neved kept track of RAM usage for making te databases. My dammit database folder is 59G
    Camille Scott
    @camillescott
    the databases can be quite memory intensive to generate; expect ~16gb for the orthodb one
    not sure what is necessary for uniref90, though i can try to find out for you
    Camille Scott
    @camillescott
    currently dammit doesn't have an option to custom tune parameters
    however, if you're willing to get a little dirty, you can tune them yourself
    if you can find your site-packages directory, you can edit the file. something like /path/to/python/lib/python2.7/site-packages/dammit
    there's a file in there called .config.json
    one group is called "last", and has an entry for "lastdb"
    you can add extra arguments to its params entry, and they will be used in the run
    there is a bunch of info about tuning here: http://last.cbrc.jp/doc/last-tuning.html
    the only extra param included by default is -w3
    this reduces memory usage
    you can up that to -w5 or more
    there are tradeoffs, explained in the last docs i linked
    hope that helps @ulrichkudahl :)
    future version will probably allow for more tuning (ie passing in your own config file)
    Matthew Oldach
    @moldach

    Hello Camille,

    I hope this message finds you well.

    I'm running into an issue with dammit annotate and hoping you can help.

    I ran dammit annotate /home/ubuntu/V10.Trinity.fasta --busco-group eukaryota --database-dir /home/ubuntu/dammit/ --full and got the following error:

    --- Checking PATH for dependencies
    
              [x] HMMER
    
              [x] Infernal
    
              [x] BLAST+
    
              [x] BUSCO
    
              [x] TransDecoder
    
              [x] LAST
    
              [x] crb-blast
    
    
    --- Dependency results
    
              All dependencies satisfied!
    
    
    --- Checking for database prep (dir: /home/ubuntu/dammit)
    
              [x] download_and_gunzip:Pfam-A.hmm
    
              [x] hmmpress:Pfam-A.hmm
    
              [x] download_and_gunzip:Rfam.cm
    
              [x] cmpress:Rfam.cm
    
              [x] download_and_gunzip:aa_seq_euk.fasta
    
              [x] lastdb:aa_seq_euk.fasta.db
    
              [x] download_and_gunzip:ODB8_EukOGs_genes_ALL_levels.txt
    
              [x] download_and_untar:dammit-eukaryota
    
              [x] download_and_gunzip:uniref90.fasta
    
              [x] lastdb:uniref90.fasta.db
    
    
    --- Database results
    
              All databases prepared!
    
    
    --- Running annotate!
    
              Transcriptome file: V10.Trinity.fasta
    
              Output directory: /home/ubuntu/dammit/V10.Trinity.fasta.dammit
    
              [x] V10.Trinity.fasta
    
              [x] transcriptome_stats:V10.Trinity.fasta
    
              [x] busco:V10.Trinity.fasta-eukaryota
    
              [x] TransDecoder.LongOrfs:V10.Trinity.fasta
    
              [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm
    
    Some tasks failed![dammit.annotate:ERROR]
    TaskError - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm[dammit.annotate:ERROR]
    Command error: 'cat V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep | None --round-robin --pipe -L 2 -N 10000 --gnu -j 1 -a V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep /usr/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /home/ubuntu/dammit/Pfam-A.hmm /dev/stdin > V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 127
    [dammit.annotate:ERROR]

    As per your earlier comments with @ulrichkudahl I tried which parallel and found that it was not installed. After sudo apt-get install parallel I re-ran the dammit annotate command and a new error was returned.

    ...
    [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm
    
    Some tasks failed![dammit.annotate:ERROR]
    TaskError - taskid:hmmscan:longest_orfs.pep.x.Pfam-A.hmm[dammit.annotate:ERROR]
    Command error: 'cat V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep | /usr/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 36 -a V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep /usr/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl.out /home/ubuntu/dammit/Pfam-A.hmm /dev/stdin > V10.Trinity.fasta.transdecoder_dir/longest_orfs.pep.pfam.tbl' returned 255
    [dammit.annotate:ERROR]

    Your help is greatly appreciated.
    Thanks,
    Matt

    Matt MacManes
    @macmanes
    Hey @moldach, which version of gnu parallel do you have installed? I remember that dammit requires a newer version than I initially had. I used Linuxbrew to install.
    Matthew Oldach
    @moldach
    Thanks for the tip @macmanes brew install parallel worked!
    b4ph0m3t
    @b4ph0m3t

    hello I am trying to run dammit over the provided sample data in a CentOS machine.
    It runs until

    
              [ ] lastal:cdna_nointrons_utrs.fa.x.orthodb.maf
    
    Some tasks failed![dammit.annotate:ERROR]
    TaskFailed - taskid:lastal:cdna_nointrons_utrs.fa.x.orthodb.maf[dammit.annotate:ERROR]
    Command failed: 'cat cdna_nointrons_utrs.fa | /usr/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 1 -a cdna_nointrons_utrs.fa "/home/labgen/Software/last-658/src/lastal -F15 -D100000.0 /home/labgen/.dammit/databases/aa_seq_euk.fasta.db" > cdna_nointrons_utrs.fa.x.orthodb.maf' returned 1
    [dammit.annotate:ERROR]

    happens.
    Is it a dependency issue?
    Thanks

    b4ph0m3t
    @b4ph0m3t

    solved the error by deleting .prj file from /databases/ folder and re-running

    dammit databases --install

    there was some strange issue with lastal reading .prj file (some lines were missing)

    Matthew Oldach
    @moldach

    It appear like dammit is working, although maybe not a 100% since not all of the boxes were checked:

    ubuntu@ip-172-31-2-75:~/dammit$ dammit annotate /home/ubuntu/good.V10.Trinity.fasta --busco-group metazoa --database-dir /home/ubuntu/dammit/ --full
    ========================================
    dammit! a tool for easy de novo transcriptome annotation
    Camille Scott 2015
    ========================================
    
    ~~~~~~~~~~~~~~~~~~~~~~~~

    submodule: annotate

    
    --- Checking PATH for dependencies
    
              [x] HMMER
    
              [x] Infernal
    
              [x] BLAST+
    
              [x] BUSCO
    
              [x] TransDecoder
    
              [x] LAST
    
              [x] crb-blast
    
    
    --- Dependency results
    
              All dependencies satisfied!
    
    
    --- Checking for database prep (dir: /home/ubuntu/dammit)
    
              [x] download_and_gunzip:Pfam-A.hmm
    
              [x] hmmpress:Pfam-A.hmm
    
              [x] download_and_gunzip:Rfam.cm
    
              [x] cmpress:Rfam.cm
    
              [x] download_and_gunzip:aa_seq_euk.fasta
    
              [x] lastdb:aa_seq_euk.fasta.db
    
              [x] download_and_gunzip:ODB8_EukOGs_genes_ALL_levels.txt
    
              [x] download_and_untar:dammit-metazoa
    
              [x] download_and_gunzip:uniref90.fasta
    
              [x] lastdb:uniref90.fasta.db
    
    
    --- Database results
    
              All databases prepared!
    
    
    --- Running annotate!
    
              Transcriptome file: good.V10.Trinity.fasta
    
              Output directory: /home/ubuntu/dammit/good.V10.Trinity.fasta.dammit
    
              [x] good.V10.Trinity.fasta
    
              [x] transcriptome_stats:good.V10.Trinity.fasta
    
              [x] busco:good.V10.Trinity.fasta-metazoa
    
              [x] TransDecoder.LongOrfs:good.V10.Trinity.fasta
    
              [x] hmmscan:longest_orfs.pep.x.Pfam-A.hmm
    
              [x] remap_hmmer:longest_orfs.pep.pfam.tbl
    
              [ ] TransDecoder.Predict:good.V10.Trinity.fasta
    
              [x] cmscan:good.V10.Trinity.fasta.x.Rfam.cm
    
              [x] lastal:good.V10.Trinity.fasta.x.orthodb.maf
    
              [x] lastal:good.V10.Trinity.fasta.x.uniref.maf
    
              [x] maf_best_hits:good.V10.Trinity.fasta.x.orthodb.maf-
              good.V10.Trinity.fasta.x.orthodb.maf.best.csv
    
              [x] maf-gff3:good.V10.Trinity.fasta.x.orthodb.maf.gff3
    
              [x] maf_best_hits:good.V10.Trinity.fasta.x.uniref.maf-
              good.V10.Trinity.fasta.x.uniref.maf.best.csv
    
              [x] maf-gff3:good.V10.Trinity.fasta.x.uniref.maf.best.csv.gff3
    
              [x] hmmscan-gff3:good.V10.Trinity.fasta.pfam.csv.gff3
    
              [x] cmscan-gff3:good.V10.Trinity.fasta.rfam.tbl.gff3
    
              [ ] gff3-merge:good.V10.Trinity.fasta.dammit.gff3
    
              [x] fasta-annotate:good.V10.Trinity.fasta.dammit.fasta

    Does this mean TransDecoder.Predict and gff3-merge failed to run?
    Any suggestions for how to force them to complete if so? I tried running the command a number of times to no avail

    Thank you

    b4ph0m3t
    @b4ph0m3t
    from my understanding checked boxes mean that the output file of the command is already present and updated so it is not necessary to execute the related command.
    not 100% sure tho
    Adam Stuckert
    @AdamStuckert
    Hi @camillescott, thanks for a great program! I looked for information on citing dammit, but I couldn't find it on your website or the github page. Help?