Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 30 07:37

    ocramz on gh-pages

    Add arrayfire (compare)

  • Jan 02 12:51

    ocramz on gh-pages

    add inliterate (compare)

  • Jan 02 12:43

    ocramz on gh-pages

    update hvega entry (compare)

  • Jul 01 2019 09:43
    dmvianna added as member
  • Jun 15 2019 04:55

    ocramz on gh-pages

    Add pcg-random (compare)

  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz opened #42
  • Jun 14 2019 16:08
    ocramz opened #42
  • Jun 06 2019 18:21

    ocramz on gh-pages

    Fix graphite link Merge pull request #41 from alx… (compare)

  • Jun 06 2019 18:21
    ocramz closed #41
  • Jun 06 2019 18:21
    ocramz closed #41
  • Jun 06 2019 17:32
    alx741 opened #41
  • Jun 06 2019 17:32
    alx741 opened #41
  • Jun 06 2019 16:46

    ocramz on gh-pages

    Add graphite Merge pull request #40 from alx… (compare)

  • Jun 06 2019 16:46
    ocramz closed #40
Yves Parès
@YPares
Hi guys, I and a few people started a room for pro/semi-pro Haskellers to discuss architecture of purely functional programs, mostly how to deal with effect composition frameworks (be it via raw monad stacks, free monads/applicatives, mtl, extensible-effects, polysemy, etc) which one is the best and interoperate between different libs written in different frameworks.
Its goal is to be "Effect composition for the rest of us": we want to have a focus on practical intermediate to advanced stuff (ie. remain accessible for those who don't know/want to know about coyoneda lemma or right-kan adjustments)
Ideal scenario: we determine the best effect composition pattern to rule them all and take over the pure FP world
Good enough scenario: we produce insights, use cases and material (or just discussions) about which way to best address effect composition in such specific scenarios and how to deal with interop between frameworks
Yves Parès
@YPares
You can join the public community room https://gitter.im/pure-fp-architects/community if you are interested, and just drop a line if you are interested. The actual room is invite only
Yves Parès
@YPares
(by "which one is the best" I meant "which one seems to be the most suited in specific contexts", I'm not intending to solve what the Haskell community as a whole has been struggling with for 20 years ^^, don't worry)
Austin Huang
@austinvhuang
@YPares do you do much ghc hacking or deeper dives into the RTS? or know anyone that might be useful to ask about the RTS?
Torsten Scholak
@tscholak
Hi, didn’t know about this channel, I see lots of familiar faces!
Tim Pierson
@o1lo01ol1o
:wave:
Marco Zocca
@ocramz
Hi @tscholak , welcome !
Austin Huang
@austinvhuang

Hi @tscholak!

Indeed... upside and downside of doing data science + machine learning in Haskell, you know everyone after about five minutes!

that's alright though, there's a strong selection bias for interesting people :)
Yves Parès
@YPares
@austinvhuang For my part, I never dabbed with GHC internals, but I think we have people at Tweag who did work on the RTS. I can ask around
Austin Huang
@austinvhuang
Thanks @YPares !
Compl Yue
@complyue
I almost succeeded in distributing custom GHC by source with Nix, but met an annoying problem, any tip is appreciated: NixOS/nixpkgs#73443
David Johnson
@dmjio
hey
data haskell community
I was talking with the author of massiv https://github.com/lehins about making some benchmarks for every Haskell array library (CPU + GPU). It seems you guys have done that already here: https://github.com/DataHaskell/numeric-libs-benchmarks
We don't know where to house a repo like this, and were wondering if we could contribute / work with people here to make a repo like this (even if that means contributing to the existing repo).
David Johnson
@dmjio
maybe this conversation would be best had on github
but like, I was gonna make all these nix scripts to build accelerate, massiv, hmatrix, arrayfire into one giant kahuna repo, then run the benchmarks on AWS, scp the results to some s3 bucket
David Johnson
@dmjio
Marco Zocca
@ocramz
hey @dmjio , when it comes to low-level array functions there's this : https://github.com/haskell-perf/sequences
the numeric-libs-benchmarks repo is very incomplete wrt fastest-matrices
Shane Mulligan
@mullikine
Hey guys. does anyone connect to this chat room using IRC? I am connected to irc.gitter.im but when I join #dataHaskell, I get an empty room
I worked it out. I had to join #dataHaskell/Lobby.
Isaac Shapira
@fresheyeball_gitlab
I am looking for some to do a small CV contract
anyone interested?
Marco Zocca
@ocramz
@fresheyeball_gitlab CV?
Isaac Shapira
@fresheyeball_gitlab
@ocramz Computer Vision
I need object detection trained, and I have a labelled data set. I have done some ML, but not any Vision stuff. Looking for a contractor to either do the work, or hold my hand.
Asking this room because long term this would be a Haskell project.
Stefan Dresselhaus
@Drezil
what do you mean by "small contract"? A thing you could do in 2-3 days? Or more like 3-4 weeks?
Compl Yue
@complyue
it's been a pleasure to learn Nix around, and I finally made it to do auto sdist install of GHC, so
comes the first unregistered release of Hadui. I'm quite satisfied with the approach I discovered, to define a package merely with github release tarballs, that directly importable by nix-shell
/
nix-env -iA. this is quite easy way to do decentralized software distribution based off nixpkgs, even without registration to its central registry, should apply equally well to internal distribution of private packages.
Compl Yue
@complyue
the 3 flavored scaffold repositories demo this technique: https://github.com/complyue/hadui/blob/master/README.md#quick-start
Orfeas Agis Karachalios
@orpheusgame
hi people, i was curious if someone has considered doing number-crunching in Julia and implement some sort of foreign calls using the LLVM IR
chessai
@chessai
Is there a library which makes the following clean:

Suppose a 2d array initialisation whereby each entry depends on the entries immediately above the current entry and to the left and above and to the left. That can naturally be computed in parallel by traversing along the diagonals. Example:

 1 2 3
 4 5 6
 7 8 9

You start with 1, then compute 4 and 2 in parallel then 7 5 3 in parallel then 8 6 in parallel then 9 in parallel

Im having trouble coming up with a way to do this with arrayfire/repa/massiv
Alexey Kuleshevich
@lehins
I had plans (still do) on creating a stencil that allows doing the waterfall parallelization. I read about it a paper about PASTHA, I believe. I didn't think people would need such functionality, so I didn't even rush on adding such thing, I guess I was wrong ;)
@chessai Doing so in a reusable (general multidimensional) fashion is certainly non trivial, but for a specific case like a 2D array it is not too hard to do with massiv, but not straightforward either. Let me think about it for a moment and I'll try to come up with an example for you.
Alexey Kuleshevich
@lehins
By the way, here is a paper that describes the method I want to implement (at least variation of it): https://dl.acm.org/citation.cfm?id=1708052&dl=ACM&coll=DL
Alexey Kuleshevich
@lehins
@chessai Here is almost something what you were looking for, in fact it is a bit better. Cause you don't want to schedule individual elements to be computed in parallel, but rather chunks of the diagonals. This won't matter for small arrays, but for bigger ones I suspect there will be difference. Anyways, here it is:
waterfallCreate ::
     (Mutable r Ix2 a, PrimMonad m, MonadUnliftIO m, MonadThrow m)
  => Sz2
  -> (Maybe a -> Maybe a -> a)
  -> (a -> a -> a)
  -> m (Array r Ix2 a)
waterfallCreate sz@(Sz2 m n) g f =
  createArray_ Par sz $ \scheduler marr -> do
    forM_ (0 ..: m) $ \i -> writeM marr (i :. 0) . g Nothing =<< A.read marr (i - 1 :. 0)
    --  ^ fill left edge
    forM_ (1 ..: n) $ \j -> do
      writeM marr (0 :. j) . (`g` Nothing) =<< A.read marr (0 :. j - 1)
      --  ^ fill the top edge
      let jIter = rangeStepSize Seq j (-1) (Sz (min (m - 1) j))
      iforSchedulerM_ scheduler jIter $ \i' -> writeF marr (i' + 1)
    forM_ (2 ..: m) $ \i -> do
      let jIter = rangeStepSize Seq (n - 1) (-1) (Sz (min (n - 1) (m - i)))
      iforSchedulerM_ scheduler jIter $ \i' -> writeF marr (i' + i)
  where
    writeF marr i j = do
      left <- readM marr (i :. j - 1)
      top <- readM marr (i - 1 :. j)
      writeM marr (i :. j) $ f left top
    {-# INLINE writeF #-}
{-# INLINE waterfallCreate #-}
Alexey Kuleshevich
@lehins
The reason for two functions g and f instead of one, because it will be more performant if you supply separate functions: one that produces elements for the borders and another one for the inside, this way function f always knows that it will get the neighbors it needs and doesn't have to check for that edge case. Here is a cool part, because you know that the whole array will be filled up, and since f and g are pure, we are guaranteed that the whole waterfallCreate is pure, so we can safely wrap it in unsafePerformIO. More over the interior read and write functions, are guaranteed not to be out of bounds, so once you are done with all the testing you can improve the performance even further by switching to unsafe* functions:
waterfallCreate ::
     Mutable r Ix2 a => Sz2 -> (Maybe a -> Maybe a -> a) -> (a -> a -> a) -> Array r Ix2 a
waterfallCreate sz@(Sz2 m n) g f =
  unsafePerformIO $
  createArray_ Par sz $ \scheduler marr -> do
    void $ write marr (0 :. 0) $ g Nothing Nothing
    forM_ (1 ..: m) $ \i ->
      unsafeWrite marr (i :. 0) . g Nothing . Just =<< unsafeRead marr (i - 1 :. 0)
    forM_ (1 ..: n) $ \j -> do
      unsafeWrite marr (0 :. j) . (`g` Nothing) . Just  =<< unsafeRead marr (0 :. j - 1)
      let jIter = rangeStepSize Seq j (-1) (Sz (min (m - 1) j))
      iforSchedulerM_ scheduler jIter $ \i' -> writeF marr (i' + 1)
    forM_ (2 ..: m) $ \i -> do
      let jIter = rangeStepSize Seq (n - 1) (-1) (Sz (min (n - 1) (m - i)))
      iforSchedulerM_ scheduler jIter $ \i' -> writeF marr (i' + i)
  where
    writeF marr i j = do
      left <- unsafeRead marr (i :. j - 1)
      top <- unsafeRead marr (i - 1 :. j)
      unsafeWrite marr (i :. j) $ f left top
    {-# INLINE writeF #-}
{-# INLINE waterfallCreate #-}
Couple examples won't hurt:
λ> g mx my = maybe (fromMaybe 0 my) (+1) mx
λ> waterfallCreate (Sz2 4 6) g (+) :: Array P Ix2 Int
Array P Par (Sz (4 :. 6))
  [ [ 0, 1, 2, 3, 4, 5 ]
  , [ 0, 1, 3, 6, 10, 15 ]
  , [ 0, 1, 4, 10, 20, 35 ]
  , [ 0, 1, 5, 15, 35, 70 ]
  ]
λ> waterfallCreate (Sz2 6 4) g (+) :: Array P Ix2 Int
Array P Par (Sz (6 :. 4))
  [ [ 0, 1, 2, 3 ]
  , [ 0, 1, 3, 6 ]
  , [ 0, 1, 4, 10 ]
  , [ 0, 1, 5, 15 ]
  , [ 0, 1, 6, 21 ]
  , [ 0, 1, 7, 28 ]
  ]
Alexey Kuleshevich
@lehins

To sum it up:

  • Not sure about arrayfire, but I kow for a certain you can't do that in repa
  • Small mistake in what you said, you can't compute 9 in parallel, since it depends on 8 and 6

    8 6 in parallel then 9 in parallel

  • For parallelization it is always better to schedule bigger tasks, rather than a bunch of small ones, that is why splitting loading of diagonals in chunks with iforSchedulerM_ is better. That being said, if you still feel like you want to load each element in parallel, you can easily do that by switching to manually doing scheduleWork_ per each element:
    iforM_ jIter $ \i' -> scheduleWork_ scheduler $ writeF marr (i' + 1)
  • Another optimization I want to do, once I get to actually adding that to massiv is try to perform loading in blocks, rather than strips of diagonals, just cause it is better for cache locality and will be faster, but that is a totally separate story ;)
chessai
@chessai
@lehins thanks! This is extremely helpful!
chessai
@chessai
Makes me excited about massiv