Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 12 17:17

    nellore on master

    adds deprecation note (compare)

  • Nov 30 2020 19:01
    Rimsk commented #85
  • Nov 30 2020 18:58
    Rimsk commented #85
  • Jul 09 2020 23:08

    nellore on master

    adds unit test to alignment_han… (compare)

  • Oct 18 2019 17:59
    muhammed-ali commented #85
  • Oct 09 2019 17:29
    dfermin commented #85
  • Oct 08 2019 13:14
    dfermin commented #85
  • Apr 09 2019 00:16

    BenLangmead on master

    embed parsed_md (compare)

  • Apr 08 2019 21:26

    BenLangmead on master

    make more portable (compare)

  • Apr 08 2019 21:18

    BenLangmead on master

    some attempts to make this scri… (compare)

  • Dec 03 2018 15:25
    gianmaz edited #88
  • Dec 03 2018 14:25
    gianmaz opened #88
  • Mar 15 2018 20:05
    ChristopherWilks opened #87
  • Mar 15 2018 20:01

    ChristopherWilks on master

    switched to use sratoolkit 2.8.… (compare)

  • Mar 04 2018 22:50

    nellore on master

    patches bowtie2-build in travis… (compare)

  • Mar 04 2018 22:11

    nellore on master

    uses bowtie2 2.3.4.1 (compare)

  • Mar 04 2018 21:57

    nellore on master

    specifies samtools version to i… (compare)

  • Mar 04 2018 21:47

    nellore on master

    updates dependencies Merge branch 'master' of https:… (compare)

  • Mar 04 2018 21:39

    nellore on master

    quote rules Merge pull request #86 from Ben… (compare)

  • Mar 04 2018 21:39
    nellore closed #86
abhinav
@nellore
thank you!!!
Vy T. Nguyen
@vnguyen0009
Quick question, which script makes the precoverage/dp.task files?
abhinav
@nellore
coverage_pre.py
Vy T. Nguyen
@vnguyen0009
Thank you!
Vy T. Nguyen
@vnguyen0009
Hello! Just running into some issues. We were able to run rail through our data sets but now we need to compile them into gene count tables. Is there an easy way of doing this? I took a look at the tutorial you put a link to on the rail homepage (http://jtleek.com/protocols/bigwig_gene_counts/) but I am having some trouble figuring out how to do it for all chromosomes.
Vy T. Nguyen
@vnguyen0009
Hello, I am still trying to obtain gene counts from my data obtained from Rail-RNA. I managed to convert the BigWig files to Wig files and was wondering if you could clarify what the values in the last column. It would really help me be able to figure out how to get to gene counts.

bedGraph section chr1:0-631636

chr1 0 14678 0
chr1 14678 14689 1
chr1 14689 14690 6
chr1 14690 14692 7
chr1 14692 14697 10
chr1 14697 14754 22
chr1 14754 14765 21
chr1 14765 14766 16
chr1 14766 14768 14
chr1 14768 14771 12
chr1 14771 14772 5
m-liddell
@m-liddell
Hi, I'm having some problems with an error on the Step 22/24: Write bigWigs with exome coverage by sample step:
00h:50m:26s |___| Step 22/24: Write bigWigs with exome coverage by sample
00h:51m:11s |___|     Partitioned 39 inputs into tasks.
*****Errors encountered*****
Streaming command "LC_ALL=C sort -S 307200 -k1,1 -k2,3 -t$'\t' -m /home/rail-rna_logs/coverage/dp.tasks/2.* | /root/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /root/raildotbio/rail-rna/rna/steps/coverage.py --bowtie-idx=/index/BowtieIndex/genome --percentile=0.75 --out=/markliddell/rail-rna/output/coverage_bigwigs --bigwig-exe=/root/raildotbio/bedGraphToBigWig --manifest=/markliddell/rail-rna/manifest.txt >/home/rail-rna_logs/coverage/2 2>/home/rail-rna_logs/coverage/dp.reduce.log/2.0.log" failed; exit level was 1.
Job flow failed on Sunday, Mar 01, 2020 at 02:14:34 PM UTC. Run time was 3073.353 seconds.
To start this job flow from where it left off, run:
/root/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /root/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /home/rail-rna_logs/resume_flow_7P2J6ET8GJ6I.json -b /root/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /home/rail-rna_logs/flow.2020-03-01T13:23:19.915920.log -f --max-attempts 1 --num-processes 39
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
    max_attempts=max_attempts
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError
m-liddell
@m-liddell
/home/rail-rna_logs/coverage/dp.reduce.log/2.0.log:
/root/raildotbio/bedGraphToBigWig: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/root/raildotbio/rail-rna/rna/steps/coverage.py", line 334, in <module>
    % bedtobigwig_process.returncode)
RuntimeError: bedgraphtobigwig process failed w/ exitlevel 127.
m-liddell
@m-liddell
Fixed it. The cause was that the image on dockhub (https://hub.docker.com/r/congelos/rail-rna) is missing libpng12-0.
sonali-bioc
@sonali-bioc
Hi , I am getting the same error as @m-liddell above : /home/sarora/raildotbio/bedGraphToBigWig: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/home/sarora/raildotbio/rail-rna/rna/steps/coverage.py", line 334, in <module>
% bedtobigwig_process.returncode)
RuntimeError: bedgraphtobigwig process failed w/ exitlevel 127. I do have libpng/1.6.37-GCCcore-8.3.0 already installed on the cluster that I am using rail-rna on.. and am loading the library before I re-start my rail-rna job. But the same error persists . How can I solve this ? Thank you!!
abhinav
@nellore
@sonali-bioc i'm hoping it's because the image on dockhub is missing libpng12-0 and just requires its installation, but we did not create that image! if that is true, can you just install rail-rna by following the instructions at rail.bio?
sonali-bioc
@sonali-bioc

Hi @nellore - thanks for looking at this quickly - I did install via following instructions on rail.bio - not using the docker image :sarora@rhino01:~/HollandLabShared/Sonali/tools$ ./install_rail-rna-0.2.4b
∀ Rail-RNA v0.2.4b Installer
Rail-RNA can be installed for all users or just the current user.

* Install for all users? [y/n]: n

The installation path /home/sarora/raildotbio already exists.

* Overwrite /home/sarora/raildotbio? [y/n]: n

Specify a different installation directory with --install-dir.
sarora@rhino01:~/HollandLabShared/Sonali/tools$ ./install_rail-rna-0.2.4b
∀ Rail-RNA v0.2.4b Installer
Rail-RNA can be installed for all users or just the current user.

* Install for all users? [y/n]: n

The installation path /home/sarora/raildotbio already exists.

* Overwrite /home/sarora/raildotbio? [y/n]: y

Installed Rail-RNA.
Installation log may be found at /home/sarora/raildotbio/rail-rna_installer.log.

Also, I do have libpng installed on the cluster at my workplace - sarora@rhino01:~/HollandLabShared/Sonali/tools$ module load libpng/1.6.37-GCCcore-8.3.0
I guess rail-rna just cant find it ?
I tried adding the path for libpng to "~/bin/rail-rna" but that didnt work either.
abhinav
@nellore
well it has to do with the external dependency
bedGraphToBigWig
and replace the bedGraphToBigWig executable in the rail-rna dir with it?
sonali-bioc
@sonali-bioc
sure - let me try that
done, replaced - and have resumed the job . fingers crossed !
sonali-bioc
@sonali-bioc
Yes!! I think that worked - It started writing bigwig files. I now see bigwig files under sample1/rail-rna_out/coverage_bigwigs
abhinav
@nellore
awesome!
sonali-bioc
@sonali-bioc
Thank you for all your help!!
abhinav
@nellore
sure thing!
genec1
@genec1

Hello! I'm trying to run rail on ubuntu and am running into some python errors.

The first was a problem with 'Url' not being defined in raildotbio/rail-rna/rna/driver/rna_config.py. I found at least two lines where Url(...) was being called while everywhere else it was ab.Url(...). The code ran after changing Url to ab.Url.

But then I ran into this second error:

Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 1205, in <module>
    ignore_missing_sra_samples=args.ignore_missing_sra_samples)
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 945, in go
    elif len(original_qnames) == 2 and original_qnames[1]:
UnboundLocalError: local variable 'original_qnames' referenced before assignment

Can you help me fix this?

genec1
@genec1
So I tried reinstalling -- perhaps I had installed an old version -- and I got further before getting to an error. Here's what I get now:
rail-rna go local --verbose  -x  hg38_bowtie/genome -m test_manifest.txt -o rail_out  -d tsv,bw,jx --force
Loading...
Checked all files listed in manifest file.
/home/gene/raildotbio/bowtie-1.1.2/bowtie:24: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Detected Bowtie 1 v1.1.2, Bowtie 2 v2.2.7, and SAMTools v1.2.1.
Launching Dooplicity runner with PyPy...

∀ Rail-RNA v0.2.4b
Started job flow on Tuesday, Aug 04, 2020 at 03:30:56 PM PDT.

~.oOo.>

00h:00m:00s |___| Step 1/24: Count lines in input files
00h:00m:00s |___|     Completed 1 task.
00h:00m:00s |___|     Deleted temporary files.
00h:00m:00s |___| Step 2/24: Assign reads to preprocessing tasks
00h:00m:00s |___|     Partitioned 1 input into tasks.
00h:00m:01s |___|     Completed 1 task.
00h:00m:01s |___|     Deleted temporary files.
00h:00m:01s |___| Step 3/24: Preprocess reads
*****Errors encountered*****
Streaming command "cat /tmp/tmp3OBUqt/0 | /home/gene/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /home/gene/raildotbio/rail-rna/rna/steps/preprocess.py --nucs-per-file=100000000 --gzip-output --push=/extra/gene/rail-rna_logs/preprocess/push --gzip-level 3   --bin-qualities   --fastq-dump-exe fastq-dump >/extra/gene/rail-rna_logs/preprocess/0 2>/extra/gene/rail-rna_logs/preprocess/dp.map.log/0.0.log" failed; exit level was 1.
Job flow failed on Tuesday, Aug 04, 2020 at 03:41:23 PM PDT. Run time was 626.573 seconds.
To start this job flow from where it left off, run:
/home/gene/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /extra/gene/rail-rna_logs/resume_flow_PIHJ9VSGP6NC.json -b /home/gene/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /extra/gene/rail-rna_logs/flow.2020-08-04T15:30:56.021901.log -f --max-attempts 1 --num-processes 27
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1795, in run_simulation
    max_attempts=max_attempts
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError
(base)
Hit "post" too early. This is still the same error, from the logs:
Range of quality values found from random sample of 10000 records is (0, 255).
Guessed Sanger encoding.
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 1205, in <module>
    ignore_missing_sra_samples=args.ignore_missing_sra_samples)
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 945, in go
    elif len(original_qnames) == 2 and original_qnames[1]:
UnboundLocalError: local variable 'original_qnames' referenced before assignment
Rimsk
@Rimsk
Hi, I am new to rail-RNA. I started using it to align my sequencing data files. I downloaded the Index files from Bowtie1 http://bowtie-bio.sourceforge.net/index.shtml and Bowtie2 http://bowtie-bio.sourceforge.net/bowtie2/index.shtml. I am using this command to align it but gets an error rail-rna go local -x /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1/ /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/ -m CCLE.manifest --log /nethome/rxk519/rail_RNA_results -d tsv,bw,jx -f --scratch temp_Rail_RNA
Loading...
Checked all files listed in manifest file.
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 659, in <module>
vdb_config_exe=args.vdb_config
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6167, in init
bed_basename=bed_basename, tsv_basename=tsv_basename)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 3906, in init
raise_runtime_error(base)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 1366, in raise_runtime_error
) if len(bases.errors) > 1 else bases.errors[0]
RuntimeError: Some Bowtie index files were not found. Check that index files with the extensions ".1.ebwt", ".rev.2.ebwt", ".3.bt2", ".1.bt2", ".rev.1.ebwt", ".2.ebwt", ".4.ebwt", ".2.bt2", ".3.ebwt", ".rev.1.bt2", ".rev.2.bt2", and ".4.bt2" exist with the basename(s) specified.
Rimsk
@Rimsk
Hi, I am new to rail-RNA. I started using it to align my sequencing data files. I downloaded the Index files from Bowtie1 http://bowtie-bio.sourceforge.net/index.shtml and Bowtie2 http://bowtie-bio.sourceforge.net/bowtie2/index.shtml. I am using this command to align it but gets an error. My Bowtie files are in the folder (/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1 )
rail-rna go local -x /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1/ /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/ -m CCLE.manifest --log /nethome/rxk519/rail_RNA_results -d tsv,bw,jx -f --scratch temp_Rail_RNA
Loading...
Checked all files listed in manifest file.
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 659, in <module>
vdb_config_exe=args.vdb_config
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6167, in init
bed_basename=bed_basename, tsv_basename=tsv_basename)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 3906, in init
raise_runtime_error(base)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 1366, in raise_runtime_error
) if len(bases.errors) > 1 else bases.errors[0]
RuntimeError: Some Bowtie index files were not found. Check that index files with the extensions ".1.ebwt", ".rev.2.ebwt", ".3.bt2", ".1.bt2", ".rev.1.ebwt", ".2.ebwt", ".4.ebwt", ".2.bt2", ".3.ebwt", ".rev.1.bt2", ".rev.2.bt2", and ".4.bt2" exist with the basename(s) specified.
Rimsk
@Rimsk
Can you please help me
Rimsk
@Rimsk
@vnguyen0009 Hi, I am having an issue running the script. I get an error. Can you please help me to resolve it.

Rail-RNA v0.2.4b
Started job flow on Tuesday, Dec 01, 2020 at 11:36:56 AM EST.

~.oOo.>

00h:00m:00s || Step 1/24: Count lines in input files
00h:00m:38s |
| Completed 1 task.
00h:00m:38s || Deleted temporary files.
00h:00m:38s |
| Step 2/24: Assign reads to preprocessing tasks
00h:00m:39s || Partitioned 1 input into tasks.
00h:00m:39s |
| Completed 1 task.
00h:00m:39s || Deleted temporary files.
00h:00m:39s |
| Step 3/24: Preprocess reads
00h:02m:07s || Completed 55 tasks.
00h:02m:07s |
| Deleted temporary files.
00h:02m:07s || Step 4/24: Align reads and segment them into readlets
00h:02m:24s |
| Partitioned 55 inputs into tasks.
*Errors encountered*
Streaming command "LC_ALL=C sort -T temp_Rail_RNA1 -S 307200 -k1,1 -t$'\t' -m /storage/rxk519/rail_RNA_results1/logs/align_reads/dp.tasks/2.* | /usr/local/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/raildotbio/rail-rna/rna/steps/align_reads.py --bowtie-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_bowtie1_ebwt/hg19 --bowtie2-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/hg19 --bowtie2-exe=/usr/local/raildotbio/bowtie2-2.2.7/bowtie2 --exon-differentials --partition-length=5000 --min-exon-size=9 --search-filter=1 --manifest=/nethome/rxk519/CCLE_1.manifest --max-readlet-size=25 --readlet-interval=4 --capping-multiplier=1.1 --gzip-level 3 --index-count 100 --tie-margin 0 --verbose --scratch temp_Rail_RNA1 --output-bam-by-chr --no-polyA -- 2>/storage/rxk519/rail_RNA_results1/logs/align_reads/dp.reduce.log/2.0.log" failed; exit level was 1.
Job flow failed on Tuesday, Dec 01, 2020 at 11:39:34 AM EST. Run time was 157.534 seconds.
To start this job flow from where it left off, run:
/usr/local/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /storage/rxk519/rail_RNA_results1/logs/resume_flow_1C6IGBF2JPM1.json -b /usr/local/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /storage/rxk519/rail_RNA_results1/logs/flow.2020-12-01T11:36:56.163343.log -f --max-attempts 1 --num-processes 55 --sort "sort -T temp_Rail_RNA1"
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
args.direct_write)
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
max_attempts=max_attempts
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
raise RuntimeError
RuntimeError

abhinav
@nellore
the file /storage/rxk519/rail_RNA_results1/logs/align_reads/dp.reduce.log/2.0.log will give some insight into why the step failed; however, we are discontinuing support for rail-rna because recount3 (which uses a STAR-based pipeline) will be released soon