samtools/htsjdk

Simon Andrews

@s-andrews

OK, thanks. I'll find another test file then and stop worrying about it :-)

Roman Valls Guimera

@brainstorm

Hello htsjdk folks @/here! I vaguely recall there was some htsjdk nightly builds (mavenized so that I could just drop the url in gradle.build)... can somebody refresh my memory? I would like to try some feature that was added ~30 days ago but there's no official release of it yet. IIRC it was some JFrog CI instance from the Broad (that I now cannot easily find nor is referenced on the htsjdk repo README.md)?

I know I can just build it locally but this is more convenient for me this way at the moment (working from an underpowered local computer right now :-S).

Roman Valls Guimera

@brainstorm

Nevermind, built it locally for now... but I would like to know if there's still a nightly builds server out there anyway ;)

Louis Bergelson

@lbergelson

@brainstorm Every push gets built and published here: https://broadinstitute.jfrog.io/broadinstitute/libs-snapshot/

We don't really advertise it because we delete them after some interval (3 months maybe? I'm not certain) and we don't want people relying on them in builds that should keep working forever.

Roman Valls Guimera

@brainstorm

Thanks Louis! Yes, makes sense, noted :)

When is the next htsjdk release scheduled? We are implementing htsget support for IGV-desktop (java) and we'd like to know when we can refer to an official .jar from Mavencentral?

mprabha2

@mprabha2

AILURE: Build failed with an exception.

Where:
Build file '/home/prabha/Documents/samtools-htsjdk-5240c40/build.gradle' line: 13
What went wrong:
An exception occurred applying plugin request [id: 'com.palantir.git-version', version: '0.11.0']

Failed to apply plugin [id 'com.palantir.git-version']
Cannot find '.git' directory
Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

BUILD FAILED in 19s
how can I fix it

Chris Norman

@cmnbroad

@mprabha2 Not sure what command you ran, but you need to run gradle with the repo clone as your working directory. Try ./gradlew shadowJar from within the repo clone directory.

mprabha2

@mprabha2

thank you

prabha@prabha-OptiPlex-9020:~/Documents/samtools-htsjdk-5240c40$ ./gradlew shadowjar --stacktrace

FAILURE: Build failed with an exception.

Where:
Build file '/home/prabha/Documents/samtools-htsjdk-5240c40/build.gradle' line: 13
What went wrong:
An exception occurred applying plugin request [id: 'com.palantir.git-version', version: '0.11.0']

Failed to apply plugin [id 'com.palantir.git-version']
Cannot find '.git' directory

I run samtools-htsjdk-5240c40 directory i dont kow

com.palantir.git-version'

Chris Norman

@cmnbroad

@mprabha2 It looks like you don’t have a .git folder in ~/Documents/samtools-htsjdk-5240c40. In order to build htsjdk from source with gradle, you need to first clone the https://github.com/samtools/htsjdk repository using git, and then run the build from there. Unfortunately you can’t build it from just the source without first cloning the repo.

mprabha2

@mprabha2

thanks it works now

benraha

@benraha

Hi all, quick question - can I translate samtools command (e.g view -F 256) to the java API's in an easy way ? I am using an implementation of genomic tools in Spark (adam) and would like to save some time by emitting some samtools commands ... Great job BTW!

Chris Norman

@cmnbroad

@benraha I don’t think there is any simple translation/mapping from samtools command line to htsjdk APIs, if thats what you mean. The translations aren't 1-1.

jonathan MERCIER

@bioinfornatics

Dear, firstly tanks for your works. I plan to use the java binding of htslib but on the project site it is wrote there is no support for BCFv2.2 . Does that mean that this feature will never come ?

Chris Norman

@cmnbroad

@bioinfornatics I don’t think we can say BCFv2.2 will never come, but AFAIK there are no short term plans to implement it. Also, FWIW, note that htsjdk isn’t a set of java bindings for htslib; rather it is a pure Java implementation that is independent of htslib.

jonathan MERCIER

@bioinfornatics

thanks @cmnbroad , I see … It is a so lot of works to maintain it
I wish you continued success

Oliver Ruebenacker

@curoli

Hello, Can I use the HTSJDK to read block-gzipped indexed VCF files from Google Cloud Storage? Thanks!

Oliver Ruebenacker

@curoli

Let me guess: the solution to my problem is probably https://github.com/googleapis/java-storage-nio

Louis Bergelson

@lbergelson

@curoli Yes, you can do random access from the gcs really easily by just specifying a gs:// path as the input.

As long as you have the nio library loaded

You need to use the functions that take Path as the input and not File.

Oliver Ruebenacker

@curoli

@lbergelson Awesome, thanks!

Abdullah El Kurdi

@abedkurdi

Hi all,
I cloned HTSJDK from github, and I am trying to compile it using gradlew. However, I am getting the following error:
java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7. How to know what Java version should I have or what gradle version should I choose to make the compilation works?
thanks!

Chris Norman

@cmnbroad

@abedkurdi You need a Java 8 SDK to build. Once you have that, run the ./gradlew script in the root of the repo, which will download the correct version of gradle. You can find this info in the “Building HTSJDK” section of the README.md.

Alex T.

@a0x8o

What version of samtools & htsjdk supports VCFv4.3?

Mark Schreiber

@markjschreiber

I want to add the ability for GATK and Picard to read from and write to S3. Looking at the HTSJDK code I get the impression that building an implementation of java.nio.file.spi.FileSystemProvider and related classes might be sufficient for at least HTSJDK. Does this sound right, are there other things I would need to do?

Chris Norman

@cmnbroad

@markjschreiber It would be great to see some progress on that. The NIO file system provider is the right place to start. The provider has to be loaded by the client (we don’t want to have dependencies on all the providers in htsjdk), and you would need to collect and propagate credentials somehow. But, note that not all code paths in htsjdk are nio-enabled. Many GATK tools can access GCS and hadoop via htsjdk because they use the nio-enabled code paths and java.nio.Path, but many (most ?) Picard tools still instantiate the java.io.File class directly using the user-provided input, which won't work with cloud uris. Also, there is a GATK PR that was started to do some of this (it doesn't necessarily take what I would say is the right approach, but you can get some clues there). broadinstitute/gatk#6698 Hope that helps.

Mark Schreiber

@markjschreiber

Thanks @cmnbroad, so if I understand it correctly implementing this will help but only for those parts of Picard/ GATK that use java.nio.path. I had been under the impression that GATK could use cloud paths for all tools, is it correct to say that only some can use cloud paths? Is there a list of tools that can (or cannot)?

Chris Norman

@cmnbroad

@markjschreiber I think at this point most GATK tools, especially the mainstream ones, can use cloud uris for their primary inputs and outputs. But there are definitely some that cannot, and some that are not fully enabled for ancillary files - for one example, some tools delegate file I/O to R or python code that is not cloud enabled. Picard has some tools that are cloud-enabled, but I suspect its probably the minority. See broadinstitute/picard#1653 for more details on the Picard part.

Mark Schreiber

@markjschreiber

I have an experimental, but functional nio package for S3 (read only at this stage). I have implemented a read ahead cache and I was wondering what would be a good size for the cache? I have it setup so that it can have n fragments of m size. When a fragment is > 50% read it triggers the async download of the next fragment. If I expect network bandwidth of ~ 100MB/s what do you think good values for n and m would be for typical GATK reading of a BAM? I have been playing with values and testing with GATKs CountReads but if someone knows of something more prescriptive it would be great.

Chris Norman

@cmnbroad

@markjschreiber FWIW, GATK currently uses a fixed, but user-configurable buffer size that defaults to 40MB for cloud inputs.

Matej Maric

@InimitableArtist

how can i import htsjdk to my java project and use it ?

Chris Norman

@cmnbroad

@InimitableArtist If you have a specific question about htsjdk, we may be able to help, but for general questions about how to use Java libraries, I’d suggest checkout out some of the many java tutorials online, or consulting stack overflow or reddit forums.

糕糕

@dtgz-sudo

Are there any examples or tutorials for htsjdk? My goal is to find the required sequence based on the bam file and generate an additional bam subfile. thank you all.

Chris Norman

@cmnbroad

@dtgz-sudo We don't provide any tutorials, but there are certainly some external resources out there that attempt to do so, if you do a search. There are a couple of simple examples in the repo. You might be able to start with this one which reads a file and then writes it back out. You’ll have to figure out how to do the filtering (not sure what you mean by "find the required sequence based on the bam”).

Louis Bergelson

@lbergelson

You could also look at some of the downstream tools that use it, like Picard or Gatk

Lars Van Cauter

@lars-vc

Hey I would like to use this project to read bai and csi files but I seem to get an error when trying to run gradlew

java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7
at org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:43)
at org.codehaus.groovy.reflection.GroovyClassValueFactory.<clinit>(GroovyClassValueFactory.java:35)

am I missing a depenency?

Louis Bergelson

@lbergelson

Huh, never seen that before.

That's when you run ./gradlew ?

It sounds like a gradle bug honestly.

What version of java are you using?

We recommend 8 or 11.

Both of which are out of date.

Where communities thrive

People

Repo info

Activity