@tscholak If you're interested, you can have a look at https://www.tweag.io/posts/2018-10-04-capability.html
The only drawback is when you want to run part of the effects (for instance if you want some local state or reader in your code, but not in your whole stack)
For that I think a polysemy approach is better suited (thats' why I developed the
reader-soup approach, because when I did it polysemy wasn't known yet)
cool, thanks for the chat!
@/all Our discussion @tscholak had me thinking. I was wondering if you guys could be interested in a chan for pro/semi-pro haskell devs (or serious enthusiasts) to specifically talk about architecturing Haskell programs, and more specifically composing effects. So a scope broader than this gitter or dataHaskell, but narrower than r/haskell or #haskell on IRC. Also I think small scale should be better, so I think it should be on invitation first to see how is goes first.
Could be on gitter, discord or slack. I envision the products of these discussions could be a wiki detailing some use cases, the patterns that were followed, some guidelines re what is the best effect lib to architecture you program in such specific case, etc.
This would facilitate the discussion around conventions and interoperability between libs and toolkits.
Could be on gitter, discord or slack. I envision the products of these discussions could be a wiki detailing some use cases, the patterns that were followed, some guidelines re what is the best effect lib to architecture you program in such specific case, etc.
This would facilitate the discussion around conventions and interoperability between libs and toolkits.
@gelisam gave a talk recently on different effect “patterns” in haskell
Okay :) I opened a lobby at https://gitter.im/pure-fp-architects/community . I invited you two to the main room
But the lobby is public, it's for people who want to manifest they are interested in the discussion. I shared the link on https://gitter.im/dataHaskell/Lobby
@YPares how difficult would it be to give porcupine distributed computing capabilities as provided by Dask, https://distributed.dask.org/en/latest/ ?
@tscholak So there's already https://github.com/tweag/funflow/tree/master/funflow-jobs that exists, but I never used it with porcupine. I added a few days ago an entry about distributed computing to the FAQ in porcupine's README
funflow-jobs is pretty rough though for the moment I think
3 replies
I want to be able to scale seamlessly between prototyping on a laptop and running it on our cluster
everything written in python...
configuration in yaml
and it’s hard to figure out why
also, it was my idea to give it a try...
the current alternative is a bash script ;)
a bash script that splits up the work, dispatches and launches jobs on the cluster
It's worth mentioning
transient again here. However, it's currently rough in terms of developer UX and the materials are not super accessible. (It's basically ContT over IO with semantics similar to ListT. However, composing all the streaming in a distributed setting is never trivial.) The best source of current information is probably via the author in the gitter: https://gitter.im/Transient-Transient-Universe-HPlay/Lobby. I know he's working on a new release.
Hi guys, I stumble upon a GHC bug which might bit you too. Happily, the workaround (once you have found it ^^) is easy:
tweag/porcupine#75
tweag/porcupine#75
@YPares I'm looking at porcupine for a couple of use cases. Is is possible to control how source data files are provided to the
PTask? Say my PTask is a foldM and I need to stream all the source files? Similarly, if the output files from the PTask are (effectfully) written at intervals from the fold, is there a reasonable way to interface with a datasink?
In one particular case, I've been using
streamly to do the concurrent transformations, is there a recommended way to purely pass those to a datasink or do I need to setup a fifo queue in STM?