Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 25 04:23

    mstksg on master

    benchmarks updated (compare)

  • Oct 25 04:22

    mstksg on master

    benchmarks updated (compare)

  • Oct 25 04:21

    mstksg on gh-pages

    Update 07-performance.html (compare)

  • Oct 25 04:16

    mstksg on gh-pages

    Update 07-performance.html (compare)

  • Sep 09 12:13
    ivanovs-4 commented #16
  • Sep 09 11:06
    jh14778 commented #16
  • Sep 09 06:18
    ocramz opened #17
  • Sep 09 06:17
    ocramz commented #16
  • Aug 28 11:54
    jh14778 opened #16
  • Jul 01 04:04

    mstksg on master

    bump changelog (compare)

  • Jul 01 04:04

    mstksg on v0.2.6.4

    (compare)

  • Jul 01 03:11
    mstksg closed #15
  • Jul 01 03:11

    mstksg on master

    Fix for ghc-8.10 Merge pull request #15 from ton… (compare)

  • Jun 30 06:12
    tonyday567 opened #15
  • Nov 05 2019 07:41

    mstksg on master

    new github workflow system (compare)

  • Nov 05 2019 07:19

    mstksg on actions

    big reset (compare)

  • Nov 05 2019 06:47

    mstksg on actions

    cache too big :( (compare)

  • Nov 05 2019 06:02

    mstksg on actions

    indentor (compare)

  • Nov 05 2019 06:01

    mstksg on actions

    dist (compare)

  • Nov 05 2019 05:46

    mstksg on actions

    use cache fork (compare)

Sam Stites
@stites
NPFL is also sort of a data-haskell project
Numerical programming in functional languages
and I think it's going to need submissions! Dominic (idontgetoutmuch) said it got a bit stressful trying to rally people to submit
say! I am running up against a bit of an API change that I think I want to encorporate into hasktorch that I think you have good insight into
Sam Stites
@stites
Primarily, I am realizing that a mutable API is nessecary and I'm wondering if there are good options which exist. I think backprop uses ST to mutate gradients inplace, is that right?
Justin Le
@mstksg
i'm going to try to submit! also heard HCAR is coming up too
hm, do you mean other libraries that manage mutation and stuff?
yeah, backprop uses ST
but most of the code is polymorphic over PrimMonad
Sam Stites
@stites
good to know! I guess I wasn't thinking of PrimMonad
Tony Day
@tonyday567
@stites I've been using generic-lens for a while now. Highly addictive.
Sam Stites
@stites
good to know!
Justin Le
@mstksg
n.b. issues with upstream deps have been resolved and backprop is back on nightlies
Justin Le
@mstksg
interestingly enough vector-sized, which a lot of my math packages rely on, is blocked on haskell/hackage-server#788
masterdezign
@masterdezign
Hello!
masterdezign
@masterdezign

I guess, there was no need to wrap data in BVar. This seems to solve the problem:

testMul :: H.L 2 2
testMul = evalBP2 (BP.<>) mk mk

mk :: H.L 2 2
mk = H.build (\a b -> a + b)

masterdezign
@masterdezign
I have another question: how do we count the number of matrix columns using Numeric.LinearAlgebra.Static? For instance, Xavier initialization depends on the number of inputs. Thank you
masterdezign
@masterdezign
To answer my previous question, I have written a not very elegant function getDim:
testMul :: H.L 2 2
testMul = evalBP2 (BP.<>) mk (H.tr mk)

mk :: H.L 2 2
mk = H.build (\a b -> a + b)

getDim :: H.Sized Double s LA.Matrix => s -> (Int, Int)
getDim s =
  let m = H.extract s :: LA.Matrix Double
  in (LA.rows m, LA.cols m)

main = do
  print $ getDim testMul
Justin Le
@mstksg
@masterdezign sorry, I have been absent from this room for a while, I hope you have been able to resolve your issues :)
if not, feel free to ask them again
masterdezign
@masterdezign
@mstksg No worries, those are resolved :)
Justin Le
@mstksg
good to hear :)
masterdezign
@masterdezign
Hello! I am playing with Accelerate LLVM PTX backend + backprop. I have a conceptual question: how do I run all the backprop computations to obtain several weight matrices? Shall I create a data structure that can be "lifted" to an equivalent of a tuple of Acc Arrays?
masterdezign
@masterdezign
  • To be more precise, I can express all neural network (fully-connected) layers as a list.
    Then, the network is:
network :: (Reifies s W) =>
=> BVar s (Acc (Matrix Float)) -- ^ Inputs 
-> BVar s [Acc (Matrix Float)]  -- ^ Weights
-> BVar s (Acc (Matrix Float))  -- ^ Outputs
Applying something like gradBP network is expected to produce [Acc (Matrix Float)], a list of gradients. Now, in gradient descent-like scenario, one would iteratively subtract that gradients from the initial weights thus obtaining a new list of weights (or equivalent). In terms of Accelerate, a GPU algorithm is constructed. To obtain the result Acc a a function run :: Arrays a => Acc a -> a is applied. How do I represent [Acc (Matrix Float)] as Arrays a => a so that I can take advantage of the backprop library?
masterdezign
@masterdezign
This is related to the issue #8
Justin Le
@mstksg
hi @masterdezign , sorry, I think I must have missed your messages earlier
i'll look at this deeper, but i think roughly the idea is to give Acc an appropriate Backprop instance
where add would be "GPU add", to provide another Acc
As for running, you might have to figure out a way to put an [Acc (Matrix Float)] as an instance of Arrays
i feel like this should be a part of the GPU library
Justin Le
@mstksg
nd how would this list/b 3
sorry, wrong window :)
Tony Day
@tonyday567
@mstksg hi Justin, I have a library, https://github.com/tonyday567/online/blob/master/online/src/Online/Averages.hs, that uses the foldl library as part of it's infrastructure. I'd like to experiment and BVar it. Do you think it's easier to ditch the foldl usage, and rewrite, or write a BVar'ed foldl?
Justin Le
@mstksg
tonyday567: hm. what function are you looking to lift to bvar?
it's interesting. it might still work as normal on one hand. but then there's also the situation with the existential type used by foldl
Tony Day
@tonyday567
all of them I suspect
Tony Day
@tonyday567
foldB :: (Reifies s W) => (BVar s Double -> BVar s Double) -> BVar s Double -> BVar s [Double] -> BVar s Double
foldB f r xs = divide (PB.foldl' (step' f r) (T2 0 0) xs) where
  step' f r (T2 s c) a = uncurry T2 ((r*) $ s + f a, (r*) $ c + 1)  
  divide (T2 s c) = s / c

stdB :: Reifies s W => BVar s Double -> BVar s [Double] -> BVar s Double
stdB r xs = (\ss s -> sqrt (ss - s ** 2)) (foldB id r xs) (foldB (\x -> x * x) r xs)
That's what it looks like unwrapped from foldl
Which is pretty sweet, but I suspect I'll lose the one-pass aspect of foldl with the applicative instance of a Fold a a
Tony Day
@tonyday567
Will try for online :: (Reifies s W, Fractional b) => (BVar s a -> BVar s b) -> (BVar s a -> BVar s b) -> Fold (BVar s a) (BVar s b)
Justin Le
@mstksg
hm. yeah, you would probably lose it in the general case
Tony Day
@tonyday567
online :: (Reifies s W, Fractional b) => (BVar s a -> BVar s b) -> (BVar s b -> BVar s b) -> Fold (BVar s a) (BVar s b)
online f g = Fold step begin extract
  where
    begin = (0, 0)
    step (s, c) a = (g $ s + f a, g $ c + 1)
    extract (s, c) = s / c

ma' :: (Reifies s W, Fractional b) => BVar s b -> Fold (BVar s b) (BVar s b)
ma' r = online id (*r)

sqma' :: (Reifies s W, Fractional b) => BVar s b -> Fold (BVar s b) (BVar s b)
sqma' r = online (\x -> x * x) (*r)

std' r = (\s ss -> sqrt (ss - s ** 2)) <$> ma' r <*> sqma' r

> backprop2 (\xs r -> L.fold (std' r) (sequenceVar xs)) [1..10::Double] 0.99
(2.8715489528256772,([-0.15247607523147438,-0.12040952611722658,-8.767961073424568e-2,-5.4276199506764516e-2,-2.0189025904650604e-2,1.4592315289823798e-2,5.007836973625318e-2,8.627982531165954e-2,0.1232075139075938,0.16087241324903193],0.14717238787841228))
Worked perfectly, how easy was that! Gradients for a moving standard deviation across [1..10] at a decay rate of 0.01.
masterdezign
@masterdezign
Hi! How do I implement heterogeneous functions such as batch normalization? During the network training I need not only gradients, but also running means. Is it the best to make BVar for all batchnorm outputs and then use noGrad selectively?
Here is an example code (no backprop package)
And here is my working branch with backprop