These are chat archives for broadinstitute/hail

5th
Aug 2016
Mitja Kurki
@Fedja
Aug 05 2016 00:10
Might this be caused by the alignment of some planets? I have 2 difference gradle versions. Running my old (weird 2.6) gradle installDist goes fine. Executing brew installed gradle 2.14 with nothing changed gives /Users/mitja/projects/hail/dev_branches/hail/src/main/scala/org/broadinstitute/hail/expr/Type.scala:80: not enough arguments for constructor AnnotationPathException: (msg: String)org.broadinstitute.hail.annotations.AnnotationPathException throw new AnnotationPathException()
Tim Poterba
@tpoterba
Aug 05 2016 00:10
gradle clean
this sometimes happens
Mitja Kurki
@Fedja
Aug 05 2016 00:11
thanks! What does it cache?
Tim Poterba
@tpoterba
Aug 05 2016 00:11
I’m not sure. It sometimes happens in either gradle or intelliJ
clean fixes gradle, rebuild fixes intelliJ
Mitja Kurki
@Fedja
Aug 05 2016 00:13
I would use \giphy voodoo now if that existed in gitter
imagine something aprropriate and funny! and thanks
Mitja Kurki
@Fedja
Aug 05 2016 15:55

halp! Why the compiler can’t infer the return type?

found : Iterator[(org.broadinstitute.hail.variant.Variant, org.broadinstitute.hail.annotations.Annotation, Iterable[T])] => Iterator[(K, V)] required: Iterator[(org.broadinstitute.hail.variant.Variant, (org.broadinstitute.hail.annotations.Annotation, Iterable[T]))] => Iterator[?] .mapPartitions { (it: Iterator[(Variant, Annotation, Iterable[T])]) =>

def aggregateByAnnotation[K,V](partitioner : Partitioner,zeroValue: V)(
    seqOp: (V, Variant, Annotation, String, Annotation, Int, T) => V,
    combOp: (V, V) => V,
    mapOp: (Variant, Annotation)  => K)(implicit uct: ClassTag[K], vct: ClassTag[V]): RDD[(K, V)] = {

    // Serialize the zero value to a byte array so that we can apply a new clone of it on each key
    val zeroBuffer = SparkEnv.get.serializer.newInstance().serialize(zeroValue)
    val zeroArray = new Array[Byte](zeroBuffer.limit)
    zeroBuffer.get(zeroArray)

    val localSampleIdsBc = sampleIdsBc
    val localSampleAnnotationsBc = sampleAnnotationsBc

    rdd
      .mapPartitions { (it: Iterator[(Variant, Annotation, Iterable[T])]) =>
        val serializer = SparkEnv.get.serializer.newInstance()
        it.map { case (v, va, gs) =>
          val zeroValue = serializer.deserialize[V](ByteBuffer.wrap(zeroArray))
          (mapOp(v,va), gs.iterator.zipWithIndex.map {
            case (g, i) => (localSampleIdsBc.value(i), localSampleAnnotationsBc.value(i),i, g) }
            .foldLeft(zeroValue) { case (acc, (s, sa, i, g)) =>
              seqOp(acc, v, va, s, sa,i, g)
            })
        }
      }
      .reduceByKey(partitioner,combOp)
  }
Tim Poterba
@tpoterba
Aug 05 2016 15:56
just explicitly type the mapPartitions call as [V]
Daniel King
@danking
Aug 05 2016 16:00
Are there any secrets to getting better error messages in IntelliJ? I've reverted to gradle compileScala. Mis-placed parentheses seem to really throw off IntelliJ.
Mitja Kurki
@Fedja
Aug 05 2016 16:01
works but why it can’t infer that in this case?
Tim Poterba
@tpoterba
Aug 05 2016 16:02
not sure
Daniel King
@danking
Aug 05 2016 18:57

So if I have a function-valued method with an implicit parameter, say:

def gen[T](sc: SparkContext, gen: VSMSubgen[T])(implicit tct: ClassTag[T]): Gen[VariantSampleMatrix[T]] =

When I try to call the returned function like this:

gen(sc, VSMSubgen.random)(someParameters)

I get an error about someParameters not being of the correct type. When I transfer the above expression to the, seemingly equivalent:

val foo = gen(sc, VSMSubgen.random)
foo(someParameters)

Everything works fine. Is there anything more elegant than adding a dummy binding and calling it?

Tim Poterba
@tpoterba
Aug 05 2016 18:59
you could use .apply in the first way, I think
gen(sc, VSMSubgen.random).apply(someParams)
Jon Bloom
@jbloom22
Aug 05 2016 18:59
yeah, hopefully that should indicate you’re not passing the implicit parameter
cseed
@cseed
Aug 05 2016 19:00
No, that should totally work. I feel like I’ve done it a ton of times.
scala> class A { def f(i: Int)(implicit j: Int): Int = i + j }
defined class A

scala> val a = new A
a: A = A@1134affc

scala> a.f(1)(2)
res1: Int = 3
Tim Poterba
@tpoterba
Aug 05 2016 19:03
what should totally work?
I don’t think that example mirrors Dan's
cseed
@cseed
Aug 05 2016 19:04
Oh, I misunderstood.
Jon Bloom
@jbloom22
Aug 05 2016 19:04
Yeah
cseed
@cseed
Aug 05 2016 19:04
He doesn’t want to pass the implicit.
Jon Bloom
@jbloom22
Aug 05 2016 19:04
Right
Daniel King
@danking
Aug 05 2016 19:04
The apply method seems to work
Sadly there's no once character alias for apply that I can find
a la Haskell's f $ 3
cseed
@cseed
Aug 05 2016 19:05
Yes, sorry, apply seems to be the common workaround.
Tim Poterba
@tpoterba
Aug 05 2016 19:08
Cotton, here’s an interesting bug in my code — vds.same(vds2) fails by finding (1:1:A:T, (Some(…), None)) and (1:1:A:T:, (None, Some(…))
if I strip the partitioners inside same with rdd.map(x => x), it works again
cseed
@cseed
Aug 05 2016 19:09
Looks like there’s a bug in your ordered join.
Tim Poterba
@tpoterba
Aug 05 2016 19:09
this isn’t using ordered join
it’s using fullOuterJoin
I think there’s a bug in the way my partitioner interacts with rdd.cogroup
cseed
@cseed
Aug 05 2016 19:10
OK, next guess: Your partitioner isn’t a real partitioner. In particular, if join gets partitioned RDDs, it’s going to choose one of the partitioners for the output, repartition the second RDD to the first RDD’s partitioner, and then do the join, shuffling only one side.
Are the partitioners the same in this case?
Tim Poterba
@tpoterba
Aug 05 2016 19:11
I think they’re both ordered, yeah
cseed
@cseed
Aug 05 2016 19:22
scala> class Rich[T](x: T) { def &:[S](f: (T) => S): S = f(x) }
defined class Rich

scala> implicit def toRich[T](x: T): Rich[T] = new Rich(x)
toRich: [T](x: T)Rich[T]

scala> (f _) &: 5
res19: Int = 6
This is probably never going to work as nicely in haskell. $ is right associative, which is indicated the trailing :. Scalaz has |> which is the same with arguments flipped.
the _ business is super annoying. Unapplied function variables are not always, immediately function values. I’m not 100% clear on the rules.
Tim Poterba
@tpoterba
Aug 05 2016 20:12
anyone care to remind me how to run a specific gradle test from the command line?
Tim Poterba
@tpoterba
Aug 05 2016 20:13
thanks jackie!