These are chat archives for twitter/summingbird

16th
May 2017
Timur
@ttim
May 16 2017 21:28
In SB we have a test in MemoryLaws - leftJoinAgainstStoreChecker. In this test we do sumByKey into store, and join with same store in independent (~parallel) node in DAG. This test enforces that sumByKey will happen firstly and leftJoin with store happen afterwards.
This test started to fail after scalacheck upgrade, I believe because functions generated by scalacheck more meaningful now. Do we really imply this law for memory platform?
/cc @pankajroark @johnynek
Pankaj Gupta
@pankajroark
May 16 2017 22:34
@ttim what do you mean by more meaningful?
Timur
@ttim
May 16 2017 22:49
@pankajroark I mean before arbitrary function generated by scalacheck was something really simple (I guess it was just constant function), and now scalacheck can generate more complex functions (which are not constant between different inputs)
Pankaj Gupta
@pankajroark
May 16 2017 23:30
Discussed offline with @ttim, this test is basically running into the same issue we had before where the test was relying on different parts on an AlsoProducer in MemoryPlatform to execute in a particular order. MemoryPlatform does not provide this guarantee but this test was passing because scalacheck was likely not testing it enough because it wasn’t generating really arbitrary values. Now scalacheck generated better arbitrary values and this test fails. Fundamentally this test seems to be relying on an incorrect guarantee. So I think we should remove this test and possibly create another test to cover what this test was meant to cover ( @ttim has some interesting arguments about why we shouldn’t do that and I will let him elaborate on that part). In a nutshell, I’m fine with removing this test to unblock summingbird scala 2.12 migration.