Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Larry N. Singh
@larryns

Hello again... I'd been analyzing a bam file with ADAM with mapping qualities that are unavailable (i.e. mapping quality = 255). I noticed that when read into an AlignmentDataset the mapping qualities corresponding to 255 are changed null, which is fine and good. The problem is when I write the values back to file with saveAsSam, the nulls are converted to 0 not 255. Any idea what I'm doing wrong, or is there a way to get 255's instead of 0?

Thanks again for all your help and for fielding my (many) questions. :)

Larry N. Singh
@larryns

P.S. my workaround right now is to :

alignmentDS.transformDataFrame(_.na.fill(255, Seq("mappingQuality"))).saveAsSam(...)

Thanks!

Michael L Heuer
@heuermh
Hello @larryns, sorry for the delay in responding! That is a bug in our conversion process, I'll file an issue and get it fixed for the next ADAM release
Larry N. Singh
@larryns
oh ok, great, thank you @heuermh! I'm glad to help
Kaustav
@kaustv-datta
Hello Team ADAM :) I'm evaluating ADAM for a new project which deals with RNA-seq data.
I wish to perform distributed deep learning on a spark cluster.
Do you have any experience with the Horovod deep learning framework?
Do you foresee any challenges in integrating Horovod with an ADAM+Spark cluster?
Michael L Heuer
@heuermh
Hello @kaustv-datta! I do not have experience with Horovod, but it appears you can integrate in at least two different ways: 1) load RNA-seq data into Spark DataFrames via ADAM and run Horovod Spark Estimator for training on those data frames; or 2) transform RNA-seq data and write out to disk as Parquet format via ADAM and run training directly on the Parquet dataset
Please let us know how you get on, it sounds like a very interesting use case!
Kaustav
@kaustv-datta
Thanks @heuermh ! I'll be sure to let you know if I do combine Horovod+ADAM
Michael L Heuer
@heuermh
:thumbsup:
Larry N. Singh
@larryns

hi, I've been searching through the documentation but couldn't find an answer so I thought I'd ask here. I apologize in advance if this question has been asked before.

I'm using a broadcastRegionJoinAgainst(gffObject) to join reads to annotations in a gff3 file. Is strand information used for determining overlap? If yes, is there a way to enforce strand matching. If no, is there a way to disable strand matching?

Thanks for your help!
Larry.

Larry N. Singh
@larryns
Any thoughts/help on my question? I've been stuck on this problem for a while. Thanks again!
Michael L Heuer
@heuermh
Hello @larryns, sorry for the delay. There was a pull request to consider strand in region joins, but it was dropped due to bit rot (bigdatagenomics/adam#1555). It should work to region join and then filter by strand afterwards.
Larry N. Singh
@larryns
Hi @heuermh , thanks for the response. I guess I'd have to do a region join, then do another join on the gff dataframe, because in some cases I need to know what strand the gene is on, especially if there are overlapping genes at the same location but on differing strands. Thanks again.
By the way, I assume by "bit rot" you mean that bugs were introduced? If there's any way I can help with this matter, please let me know. Thanks again!
Michael L Heuer
@heuermh
Oh by bit rot I mean that too much time has passed, the pull request no longer applies cleanly, the author has moved on to other things, etc.
If you have time to look at what is there, to see if it works for your use case, I should be able to help bring the changes up to date.
Larry N. Singh
@larryns
Ok, little swamped these days, but I'll take a look, maybe I can come up with a solution. Thanks!
Larry N. Singh
@larryns
hi, I'd like to write a scala app to load a bam with Adam and call variants generating a dataframe (or something similar). I'm assuming I can call avocado (or other builtin variant caller?) with an api call. Is that possible, and if so, can someone point me to how or the documentation as to how?
Larry N. Singh
@larryns
Alternatively, I'd like to do something along the lines of samtools/bcftools mpileup. As I understand, there used to be an ADAM mpileup but it was removed. I need something similar to the output of mpileup but with the read names and differences based on strand, and as I understand bcftools doesn't have that functionality. Either way, I want to process read by read (or fragment by fragment) as a stream, as opposed to creating a temporary file and then loading the file. Hopefully, that makes sense. So far the best I can think of is to create MD tags with computeMIsmatchingPositions, and then parse the MD tags, but I was hoping to avoid writing a parsers. I hope that makes sense. Thanks for any pointers.
FriederikeHanssen
@FriederikeHanssen

Hi! I have recently tried to use ADAM for duplicate marking, instead of GATK MarkDuplicates. my input files are in cram format and I keep running into this issue: 

A valid CRAM reference was not supplied and one cannot be acquired via the property settings samjdk..reference_fasta or samjdk..use_cram_ref_download

Howver, the reference is supplied. It is the same as the one that was used to create the cram files and when I use it to run GATK MarkDuplicates it works. So I am thinking, it shouldn't be the reference. Has anyone run into this or any idea what could cause this? Here is the command I run: 

adam-submit \
       --master local[*] \
       --conf spark.local.dir=. \
       --driver-memory ${task.memory.toGiga()}g \
       -- \
       transformAlignments \
       -mark_duplicate_reads \
       -single \
       -stringency LENIENT \
       -reference ${reference} \
       -sort_by_reference_position \
       ${cram} \
       ${cram.simpleName}.adam.md.cram
Michael L Heuer
@heuermh
Hello @FriederikeHanssen! I found this in the unit tests
https://github.com/bigdatagenomics/adam/blob/master/adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala#L120
It might be this property is not set when running via the command line, will check
Doesn't appear so, the -reference argument for transformAlignments is used for indel realignment, not for reading CRAM format, and htsjdk isn't smart enough to find the reference on its own
Michael L Heuer
@heuermh
bigdatagenomics/adam#2294
Michael L Heuer
@heuermh
bigdatagenomics/cannoli#288
FriederikeHanssen
@FriederikeHanssen
:thumbsup: Great, so it will be workig in the next release, right? :tada:
Michael L Heuer
@heuermh
Hopefully! I'll be testing those pull requests on AWS EMR today
Michael L Heuer
@heuermh
@FriederikeHanssen For your use case, you would be accessing CRAM files staged by Nextflow and mounted on Singularity, correct? How would the CRAM reference (.fa and .fai) be accessed?
In other words, would all of those be on local disk?
FriederikeHanssen
@FriederikeHanssen
@heuermh the cram will be staged per nextflow. The reference files are mounted via nfsmount
Larry N. Singh
@larryns
I'm having some trouble with the python version of adam. The latest version I can pull from pip is 0.31.0. Is the 0.33.0 version not available for python? Thanks.
Michael L Heuer
@heuermh
@larryns Yes, the release process for python and R has run into issues lately. I thought only 0.33.0 was missing from pypi but it appears 0.32.0 is missing as well. I am trying to get 0.33.0 up this week, and if successful, I'll go back and push 0.32.0.
Note the upcoming 0.34.0 release will have some incompatible changes due to bigdatagenomics/adam#2296 and fixes for bigdatagenomics/adam#2134 and bigdatagenomics/adam#2171
Larry N. Singh
@larryns
@heuermh okay, thank you!
Michael L Heuer
@heuermh
@larryns Version 0.33.0 was successfully pushed to pypi and a colleague says it checks out, hope it works for you!
Larry N. Singh
@larryns
@heuermh much thanks! Will let you know if I have issues.
Tanveer Ahmad
@tahashmi
Does ADAM have some API to convert a Spark Dataframe to SAM/BAM format file? Thanks.
Tanveer Ahmad
@tahashmi

Does ADAM have some API to convert a Spark Dataframe to SAM/BAM format file? Thanks.

Supposing my Dataframe has SAM data.

somiron
@somiron
Hi everyone. Does anyone know if it's possible to use HDFS for the CRAM reference genome set through hadoopbam.cram.reference-source-path ?
Larry N. Singh
@larryns
@tahashmi Something along the lines of the following should work:
// Given an adamcontext variable: ac, and a dataframe: df and that df is in the right format.
ac.loadAlignments(df)
    .saveAsSam("filename.bam", asType=Some(SAMFormat.BAM), asSingleFile=true)
Larry N. Singh
@larryns
Note that this will only work if your dataframe can be converted properly to AlignmentDataset. If you're using the htsjdk.samtools.SAMRecord objects, you can convert these to Alignment records with the SAMRecordConverter class. Of course, this approach assumes that you're using the scala API. I'm not an expert by any means, but this is the approach I would take.
Michael L Heuer
@heuermh
@tahashmi @larryns Yes, this is correct, your dataframe should have a schema (column names) as close to that of AlignmentDataset as possible for this to work. You can use e.g. Spark SQL to view/remap columns as necessary.

@somiron I have had some trouble with this in the past (bigdatagenomics/adam#1993), I haven't tried recently. Part of the problem is that Hadoop-BAM is essentially a dead project.

It is possible to use Disq to load CRAM and then convert to ADAM format, though the code in this particular repo took some classpath juggling to get to work:
https://github.com/heuermh/benchmarks/blob/master/cram/convert_cram_disq_adam.scala

somiron
@somiron
Thanks for the reply @heuermh . How do I then use the reference genome in ADAM format?
Michael L Heuer
@heuermh
CRAM references on HDFS are supposed to be supported by Hadoop-BAM, so it may work. I was never able to identify what was causing the error I saw above. You may also make the CRAM reference available on local disk
somiron
@somiron
The documentation for Hadoop-BAM is indeed saying that referencing hdfs should work. I've checked the source code and I don't see support for HDFS implemented. In the end, I copied the reference genome to HDFS and used adoop-fuse-dfs to mount HDFS locally. This seems to work fine.
Michael L Heuer
@heuermh
Good to know, thanks! It would be nice to replace Hadoop-BAM with Disq entirely at some point, but I am not convinced it would be worth the effort. I imagine rather Disq --> ADAM conversions will be added via a new module that looks something like the benchmarking scripts above.
Tanveer Ahmad
@tahashmi
Thank you so much @larryns and @heuermh
Michael L Heuer
@heuermh
FYI, I've created such a new module for using Disq to load BAM, CRAM, and VCF
https://github.com/heuermh/disq-adam
I'm not sure yet where best it should live, in the adam repository, in another repo under the bigdatagenomics organization, or in another repo under the disq-bio organization.
It requires bigdatagenomics/adam#2305