nellore/rail

Scalable RNA-seq analysis software

abhinav

@nellore

the line that reads chr2;23995 000119979136 5 1 1chr17;8056 000040283230 7 1 -1

insert a newline between 1 and chr17

then resume the pipeline

it will work

this is an I/O bug that is likely platform-specific

Vy T. Nguyen

@vnguyen0009

Okay thank you! I will do that and let you know what happens!

abhinav

@nellore

great!

Vy T. Nguyen

@vnguyen0009

which platform are you talking about, if you don't mind me asking. My Ubuntu instance?

And if I run into this issue again, would it be the same fix?

Vy T. Nguyen

@vnguyen0009

Whenever I resume flow from the run command give to me in the flow.2019-10-30T13\:56\:17.966831.log, the line I edited goes back to the way it started.

abhinav

@nellore

wow

i'll have to take another look in the next couple days

sorry you're experiencing this issue

Vy T. Nguyen

@vnguyen0009

It's all good! Thanks for taking the time. We'll be trying to do our own troubleshoot so hopefully we can figure it out together!

abhinav

@nellore

thank you!!!

Vy T. Nguyen

@vnguyen0009

Quick question, which script makes the precoverage/dp.task files?

abhinav

@nellore

coverage_pre.py

Vy T. Nguyen

@vnguyen0009

Thank you!

Vy T. Nguyen

@vnguyen0009

Hello! Just running into some issues. We were able to run rail through our data sets but now we need to compile them into gene count tables. Is there an easy way of doing this? I took a look at the tutorial you put a link to on the rail homepage (http://jtleek.com/protocols/bigwig_gene_counts/) but I am having some trouble figuring out how to do it for all chromosomes.

Vy T. Nguyen

@vnguyen0009

Hello, I am still trying to obtain gene counts from my data obtained from Rail-RNA. I managed to convert the BigWig files to Wig files and was wondering if you could clarify what the values in the last column. It would really help me be able to figure out how to get to gene counts.

bedGraph section chr1:0-631636

chr1 0 14678 0
chr1 14678 14689 1
chr1 14689 14690 6
chr1 14690 14692 7
chr1 14692 14697 10
chr1 14697 14754 22
chr1 14754 14765 21
chr1 14765 14766 16
chr1 14766 14768 14
chr1 14768 14771 12
chr1 14771 14772 5

m-liddell

@m-liddell

Hi, I'm having some problems with an error on the Step 22/24: Write bigWigs with exome coverage by sample step:

00h:50m:26s |___| Step 22/24: Write bigWigs with exome coverage by sample
00h:51m:11s |___|     Partitioned 39 inputs into tasks.
*****Errors encountered*****
Streaming command "LC_ALL=C sort -S 307200 -k1,1 -k2,3 -t$'\t' -m /home/rail-rna_logs/coverage/dp.tasks/2.* | /root/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /root/raildotbio/rail-rna/rna/steps/coverage.py --bowtie-idx=/index/BowtieIndex/genome --percentile=0.75 --out=/markliddell/rail-rna/output/coverage_bigwigs --bigwig-exe=/root/raildotbio/bedGraphToBigWig --manifest=/markliddell/rail-rna/manifest.txt >/home/rail-rna_logs/coverage/2 2>/home/rail-rna_logs/coverage/dp.reduce.log/2.0.log" failed; exit level was 1.
Job flow failed on Sunday, Mar 01, 2020 at 02:14:34 PM UTC. Run time was 3073.353 seconds.
To start this job flow from where it left off, run:
/root/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /root/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /home/rail-rna_logs/resume_flow_7P2J6ET8GJ6I.json -b /root/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /home/rail-rna_logs/flow.2020-03-01T13:23:19.915920.log -f --max-attempts 1 --num-processes 39
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
    max_attempts=max_attempts
  File "/root/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError

m-liddell

@m-liddell

/home/rail-rna_logs/coverage/dp.reduce.log/2.0.log:

/root/raildotbio/bedGraphToBigWig: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/root/raildotbio/rail-rna/rna/steps/coverage.py", line 334, in <module>
    % bedtobigwig_process.returncode)
RuntimeError: bedgraphtobigwig process failed w/ exitlevel 127.

m-liddell

@m-liddell

Fixed it. The cause was that the image on dockhub (https://hub.docker.com/r/congelos/rail-rna) is missing libpng12-0.

sonali-bioc

@sonali-bioc

Hi , I am getting the same error as @m-liddell above : /home/sarora/raildotbio/bedGraphToBigWig: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/home/sarora/raildotbio/rail-rna/rna/steps/coverage.py", line 334, in <module>
% bedtobigwig_process.returncode)
RuntimeError: bedgraphtobigwig process failed w/ exitlevel 127. I do have libpng/1.6.37-GCCcore-8.3.0 already installed on the cluster that I am using rail-rna on.. and am loading the library before I re-start my rail-rna job. But the same error persists . How can I solve this ? Thank you!!

abhinav

@nellore

@sonali-bioc i'm hoping it's because the image on dockhub is missing libpng12-0 and just requires its installation, but we did not create that image! if that is true, can you just install rail-rna by following the instructions at rail.bio?

sonali-bioc

@sonali-bioc

Hi @nellore - thanks for looking at this quickly - I did install via following instructions on rail.bio - not using the docker image :sarora@rhino01:~/HollandLabShared/Sonali/tools$ ./install_rail-rna-0.2.4b
∀ Rail-RNA v0.2.4b Installer
Rail-RNA can be installed for all users or just the current user.

* Install for all users? [y/n]: n

The installation path /home/sarora/raildotbio already exists.

* Overwrite /home/sarora/raildotbio? [y/n]: n

Specify a different installation directory with --install-dir.
sarora@rhino01:~/HollandLabShared/Sonali/tools$ ./install_rail-rna-0.2.4b
∀ Rail-RNA v0.2.4b Installer
Rail-RNA can be installed for all users or just the current user.

* Install for all users? [y/n]: n

The installation path /home/sarora/raildotbio already exists.

* Overwrite /home/sarora/raildotbio? [y/n]: y

Installed Rail-RNA.
Installation log may be found at /home/sarora/raildotbio/rail-rna_installer.log.

Also, I do have libpng installed on the cluster at my workplace - sarora@rhino01:~/HollandLabShared/Sonali/tools$ module load libpng/1.6.37-GCCcore-8.3.0

I guess rail-rna just cant find it ?

I tried adding the path for libpng to "~/bin/rail-rna" but that didnt work either.

abhinav

@nellore

well it has to do with the external dependency

bedGraphToBigWig

can you download it from http://hgdownload.soe.ucsc.edu/downloads.html#utilities_downloads

and replace the bedGraphToBigWig executable in the rail-rna dir with it?

sonali-bioc

@sonali-bioc

sure - let me try that

done, replaced - and have resumed the job . fingers crossed !

sonali-bioc

@sonali-bioc

Yes!! I think that worked - It started writing bigwig files. I now see bigwig files under sample1/rail-rna_out/coverage_bigwigs

abhinav

@nellore

awesome!

sonali-bioc

@sonali-bioc

Thank you for all your help!!

abhinav

@nellore

sure thing!

genec1

@genec1

Hello! I'm trying to run rail on ubuntu and am running into some python errors.

The first was a problem with 'Url' not being defined in raildotbio/rail-rna/rna/driver/rna_config.py. I found at least two lines where Url(...) was being called while everywhere else it was ab.Url(...). The code ran after changing Url to ab.Url.

But then I ran into this second error:

Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 1205, in <module>
    ignore_missing_sra_samples=args.ignore_missing_sra_samples)
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 945, in go
    elif len(original_qnames) == 2 and original_qnames[1]:
UnboundLocalError: local variable 'original_qnames' referenced before assignment

Can you help me fix this?

genec1

@genec1

So I tried reinstalling -- perhaps I had installed an old version -- and I got further before getting to an error. Here's what I get now:

rail-rna go local --verbose  -x  hg38_bowtie/genome -m test_manifest.txt -o rail_out  -d tsv,bw,jx --force
Loading...
Checked all files listed in manifest file.
/home/gene/raildotbio/bowtie-1.1.2/bowtie:24: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Detected Bowtie 1 v1.1.2, Bowtie 2 v2.2.7, and SAMTools v1.2.1.
Launching Dooplicity runner with PyPy...

∀ Rail-RNA v0.2.4b
Started job flow on Tuesday, Aug 04, 2020 at 03:30:56 PM PDT.

~.oOo.>

00h:00m:00s |___| Step 1/24: Count lines in input files
00h:00m:00s |___|     Completed 1 task.
00h:00m:00s |___|     Deleted temporary files.
00h:00m:00s |___| Step 2/24: Assign reads to preprocessing tasks
00h:00m:00s |___|     Partitioned 1 input into tasks.
00h:00m:01s |___|     Completed 1 task.
00h:00m:01s |___|     Deleted temporary files.
00h:00m:01s |___| Step 3/24: Preprocess reads
*****Errors encountered*****
Streaming command "cat /tmp/tmp3OBUqt/0 | /home/gene/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /home/gene/raildotbio/rail-rna/rna/steps/preprocess.py --nucs-per-file=100000000 --gzip-output --push=/extra/gene/rail-rna_logs/preprocess/push --gzip-level 3   --bin-qualities   --fastq-dump-exe fastq-dump >/extra/gene/rail-rna_logs/preprocess/0 2>/extra/gene/rail-rna_logs/preprocess/dp.map.log/0.0.log" failed; exit level was 1.
Job flow failed on Tuesday, Aug 04, 2020 at 03:41:23 PM PDT. Run time was 626.573 seconds.
To start this job flow from where it left off, run:
/home/gene/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /extra/gene/rail-rna_logs/resume_flow_PIHJ9VSGP6NC.json -b /home/gene/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /extra/gene/rail-rna_logs/flow.2020-08-04T15:30:56.021901.log -f --max-attempts 1 --num-processes 27
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
    args.direct_write)
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1795, in run_simulation
    max_attempts=max_attempts
  File "/home/gene/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
    raise RuntimeError
RuntimeError
(base)

Hit "post" too early. This is still the same error, from the logs:

Range of quality values found from random sample of 10000 records is (0, 255).
Guessed Sanger encoding.
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 1205, in <module>
    ignore_missing_sra_samples=args.ignore_missing_sra_samples)
  File "/home/gene/raildotbio/rail-rna/rna/steps/preprocess.py", line 945, in go
    elif len(original_qnames) == 2 and original_qnames[1]:
UnboundLocalError: local variable 'original_qnames' referenced before assignment

Rimsk

@Rimsk

Hi, I am new to rail-RNA. I started using it to align my sequencing data files. I downloaded the Index files from Bowtie1 http://bowtie-bio.sourceforge.net/index.shtml and Bowtie2 http://bowtie-bio.sourceforge.net/bowtie2/index.shtml. I am using this command to align it but gets an error rail-rna go local -x /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1/ /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/ -m CCLE.manifest --log /nethome/rxk519/rail_RNA_results -d tsv,bw,jx -f --scratch temp_Rail_RNA
Loading...
Checked all files listed in manifest file.
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 659, in <module>
vdb_config_exe=args.vdb_config
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6167, in init
bed_basename=bed_basename, tsv_basename=tsv_basename)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 3906, in init
raise_runtime_error(base)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 1366, in raise_runtime_error
) if len(bases.errors) > 1 else bases.errors[0]
RuntimeError: Some Bowtie index files were not found. Check that index files with the extensions ".1.ebwt", ".rev.2.ebwt", ".3.bt2", ".1.bt2", ".rev.1.ebwt", ".2.ebwt", ".4.ebwt", ".2.bt2", ".3.ebwt", ".rev.1.bt2", ".rev.2.bt2", and ".4.bt2" exist with the basename(s) specified.

Rimsk

@Rimsk

Hi, I am new to rail-RNA. I started using it to align my sequencing data files. I downloaded the Index files from Bowtie1 http://bowtie-bio.sourceforge.net/index.shtml and Bowtie2 http://bowtie-bio.sourceforge.net/bowtie2/index.shtml. I am using this command to align it but gets an error. My Bowtie files are in the folder (/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1 )
rail-rna go local -x /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie1/ /projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/ -m CCLE.manifest --log /nethome/rxk519/rail_RNA_results -d tsv,bw,jx -f --scratch temp_Rail_RNA
Loading...
Checked all files listed in manifest file.
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/raildotbio/rail-rna/main.py", line 659, in <module>
vdb_config_exe=args.vdb_config
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 6167, in init
bed_basename=bed_basename, tsv_basename=tsv_basename)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 3906, in init
raise_runtime_error(base)
File "/usr/local/raildotbio/rail-rna/rna/driver/rna_config.py", line 1366, in raise_runtime_error
) if len(bases.errors) > 1 else bases.errors[0]
RuntimeError: Some Bowtie index files were not found. Check that index files with the extensions ".1.ebwt", ".rev.2.ebwt", ".3.bt2", ".1.bt2", ".rev.1.ebwt", ".2.ebwt", ".4.ebwt", ".2.bt2", ".3.ebwt", ".rev.1.bt2", ".rev.2.bt2", and ".4.bt2" exist with the basename(s) specified.

Rimsk

@Rimsk

Can you please help me

Rimsk

@Rimsk

@vnguyen0009 Hi, I am having an issue running the script. I get an error. Can you please help me to resolve it.

Rail-RNA v0.2.4b
Started job flow on Tuesday, Dec 01, 2020 at 11:36:56 AM EST.

~.oOo.>

00h:00m:00s || Step 1/24: Count lines in input files
00h:00m:38s || Completed 1 task.
00h:00m:38s || Deleted temporary files.
00h:00m:38s || Step 2/24: Assign reads to preprocessing tasks
00h:00m:39s || Partitioned 1 input into tasks.
00h:00m:39s || Completed 1 task.
00h:00m:39s || Deleted temporary files.
00h:00m:39s || Step 3/24: Preprocess reads
00h:02m:07s || Completed 55 tasks.
00h:02m:07s || Deleted temporary files.
00h:02m:07s || Step 4/24: Align reads and segment them into readlets
00h:02m:24s || Partitioned 55 inputs into tasks.
*Errors encountered*
Streaming command "LC_ALL=C sort -T temp_Rail_RNA1 -S 307200 -k1,1 -t$'\t' -m /storage/rxk519/rail_RNA_results1/logs/align_reads/dp.tasks/2.* | /usr/local/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/raildotbio/rail-rna/rna/steps/align_reads.py --bowtie-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_bowtie1_ebwt/hg19 --bowtie2-idx=/projects/ccs/schurerlab/Rimpi/Genomes/BowtieForRailRNA/hg19_Bowtie2/hg19 --bowtie2-exe=/usr/local/raildotbio/bowtie2-2.2.7/bowtie2 --exon-differentials --partition-length=5000 --min-exon-size=9 --search-filter=1 --manifest=/nethome/rxk519/CCLE_1.manifest --max-readlet-size=25 --readlet-interval=4 --capping-multiplier=1.1 --gzip-level 3 --index-count 100 --tie-margin 0 --verbose --scratch temp_Rail_RNA1 --output-bam-by-chr --no-polyA -- 2>/storage/rxk519/rail_RNA_results1/logs/align_reads/dp.reduce.log/2.0.log" failed; exit level was 1.
Job flow failed on Tuesday, Dec 01, 2020 at 11:39:34 AM EST. Run time was 157.534 seconds.
To start this job flow from where it left off, run:
/usr/local/raildotbio/pypy-2.5-linux_x86_64-portable/bin/pypy /usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py -j /storage/rxk519/rail_RNA_results1/logs/resume_flow_1C6IGBF2JPM1.json -b /usr/local/raildotbio/rail-rna/rna/driver/rail-rna.txt -l /storage/rxk519/rail_RNA_results1/logs/flow.2020-12-01T11:36:56.163343.log -f --max-attempts 1 --num-processes 55 --sort "sort -T temp_Rail_RNA1"
Traceback (most recent call last):
File "app_main.py", line 75, in run_toplevel
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 2044, in <module>
args.direct_write)
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1900, in run_simulation
max_attempts=max_attempts
File "/usr/local/raildotbio/rail-rna/dooplicity/emr_simulator.py", line 1359, in execute_balanced_job_with_retries
raise RuntimeError
RuntimeError

abhinav

@nellore

the file /storage/rxk519/rail_RNA_results1/logs/align_reads/dp.reduce.log/2.0.log will give some insight into why the step failed; however, we are discontinuing support for rail-rna because recount3 (which uses a STAR-based pipeline) will be released soon

Where communities thrive

People

Repo info

Activity

bedGraph section chr1:0-631636