Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 20 05:04

    ocramz on gh-pages

    Add `sampling` (compare)

  • May 19 09:03

    ocramz on gh-pages

    Add kdt, Supervised Learning se… (compare)

  • Apr 14 01:32
    tonyday567 removed as member
  • Jan 30 07:37

    ocramz on gh-pages

    Add arrayfire (compare)

  • Jan 02 12:51

    ocramz on gh-pages

    add inliterate (compare)

  • Jan 02 12:43

    ocramz on gh-pages

    update hvega entry (compare)

  • Jul 01 2019 09:43
    dmvianna added as member
  • Jun 15 2019 04:55

    ocramz on gh-pages

    Add pcg-random (compare)

  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz labeled #42
  • Jun 14 2019 16:08
    ocramz opened #42
  • Jun 14 2019 16:08
    ocramz opened #42
  • Jun 06 2019 18:21

    ocramz on gh-pages

    Fix graphite link Merge pull request #41 from alx… (compare)

  • Jun 06 2019 18:21
    ocramz closed #41
  • Jun 06 2019 18:21
    ocramz closed #41
  • Jun 06 2019 17:32
    alx741 opened #41
Bryan Buecking
@buecking
@lehins do you know of any talks on youtube (or the like) that dives into porcupine? (OT. colab is the one i"m exploring now, and so far is a good fit since I"ve got some tool already for munging google's sheets)
Alexey Kuleshevich
@lehins
@buecking Here is a talk from last week at Haskell Exchange: https://skillsmatter.com/skillscasts/14236-porcupine-flows-your-rows-with-arrows
You'll probably have to register on the site to watch it, but it's free.
Bryan Buecking
@buecking
@lehins thanks - is porcupine a good fit for feed distribution in realtime?
IT looks like a very nice framework for data fundging - i'll def spend some time poking around.. i guess now i need something for visualization and backtesting
Alexey Kuleshevich
@lehins
Have no clue, haven't used before. You should ask @YPares that question, since he's the author.
Bryan Buecking
@buecking
thanks
How interesting would it be to bring Weld to Haskell?
It seems like it would cover a large number of use case for data science
Maciej J
@mjarosie

Hi all,

I'm new to both open source contributing and Haskell and I thought trying to contribute would be the best way to learn! I'm trying to resolve this issue: DataHaskell/dh-core#46 which requires using some kind of tool for uploading coverage results to codecov.io. The only one that I've found (which is mentioned in the issue description itself and on codecov's official website) is the codecov-haskell package which looks like it's not maintained anymore and doesn't compile with the newer releases of GHC (unless I'm doing something wrong). Does anyone know of a Haskell project which successfully uploads coverage reports to codecov so I could get inspired on how to deal with this?

Many thanks!
Maciek

Dmitry Pogodin
@lunaticare

Hello @mjarosie ,

I did some work on DataHaskell/dh-core#46 .
Haven't found any working alternatives for codecov-haskell, and looks like test coverage reports are not widely used by Haskell developers.
First I patched codecov-haskell library lunaticare/codecov-haskell#1 to work with recent GHC versions, had to drop older ones.
Also implemented sending reports lunaticare/dh-core#1 but the solution has several problems:

  • build takes a long time, more than 50 minutes Travis job limit, because it builds both codecov-haskell and dh-core
  • codecov-haskell works well with Cabal reports. Stack generates reports that are either different or incomplete, so after building dh-core I run tests again via Cabal and recombine reports, see test_coverage.sh file in PR

The task happened to be a bit more complex than a newbie like me could handle :)

Austin Huang
@austinvhuang

@YPares I don't have a strong opinions on the API, but I would consider what's been tried with SQL EDSLs as prior art (esquelito, persistent, opaleye etc).

In dynamicland I do like dplyr's dead-simple composition of DataFrame -> Dataframe functions (moreso than pandas's mix of method chaining and indexed operations and the higher-cognitive-overhead of Hadley's older split-apply-combine paradigm), but it's not obvious how that looks in typed form.

Maciej J
@mjarosie
Thanks @lunaticare! I'll need to figure something out! The best solution would be to take over the codecov-haskell package and make the stack-generated reports be understandable by the package as well, right?
Yves Parès
@YPares
@buecking No what I meant is that currently, given your requirements, porcupine isn't likely to help (it tries to address other problems) :)
Yves Parès
@YPares
@buecking For visualisation => http://hackage.haskell.org/package/hvega
And if by "after streaming" you mean analytics, I recommend again foldl, foldl-transduce (if some grouping is needed) and foldl-statistics
Dmitry Pogodin
@lunaticare
@mjarosie yes, I think we should make work reporting with Stack. There is a reported bug in Stack that causes it to generate incomplete reports, maybe it could be fixed.
Also, dh-core structured as a monorepo causes problems, I patched paths to source files in reports before uploading so Codecov could match code and related coverage.
Doug Burke
@DougBurke
@buecking one thing that http://hackage.haskell.org/package/hvega currently doesn't do is provide support for updating a visualization (e.g. adding points). Vega and Vega-Lite do support this, but I haven't spent any time thinking about whether it makes sense to add support in hvega. So if you want updateable graphics (rather than just re-create the whole thing with new data) you'd have to start sending JSON to whatever vega viewer you are using manually.
Compl Yue
@complyue
https://github.com/complyue/hadui I don't think it qualifies for data science before I integrated BokehJS into it, I'm going to. But before that, I'd like to share here to see if anyone interested?
Marco Z
@ocramz
@complyue wow, looks very interesting! could you describe how it works?
Compl Yue
@complyue
@ocramz um, each web page establishes a life-long websocket connection to the backend server process(es), every text packet sent from web page via the ws connection is executed dynamically with GHC API; the backend implements a (2 actually for pub/dev modes respectively) GHC frontend plugin, and managed to replace GHCi's terminal UI based repl, to interact with the websocket connection instead of stdin/stdout
Compl Yue
@complyue
hadui is tightly coupled with stack, all ways go finally invoking stack ghci against the stack project in context. it relies on stack's ecosystem to configure what's in scope in dynamic execution of web page submitted statements.
Compl Yue
@complyue
and Haskell code inside the project alter the web UI by sending text packets in json format back through the websocket conection, see https://github.com/complyue/hadui/blob/b1924e2be62a8256dc719b6b14c965b6da726041/hadui/src/HaduiRT.hs#L117 and the json cmd is interpreted at web page here: https://github.com/complyue/hadui/blob/b1924e2be62a8256dc719b6b14c965b6da726041/hadui/web/wsc.js#L122
Compl Yue
@complyue

I did some updates to the home page, added rating examples and brief:
Orientation
hadui is data science oriented, it is not suitable as a general purpose web framework.

all exported functions from all modules in the stack project of matter, are exposed to frontend in a flat space. this is ideal to support analytical workflows, but overly open or even prohibitive to support business workflows.

Platform Support
macOS - regularly used on Mojave
Linux - regularly used on Ubuntu 18.04
Windows - should work in Docker in theory, not attempted yet
GHC
currently you have to be comfortable to compile yourself an experimental version of GHC 8.6.5 with :frontend cmd to start using hadui.

do this trick to incorporate it into your stack's GHC installation.

the mod to GHC is very light, should be no difficulty to migrate to other GHC versions, but as time being, not attempted yet. a MR to GHC is thought of but not carried out yet.

Tony Day
@tonyday567
@complyue looked promising until I read the "experimental" ghc compilation step. gulp Is this a significant technical workaround or for development convenience?
Tony Day
@tonyday567
I'm currently playing around with interactive charts, using websockets via javascript-bridge. A natural development direction would be to hook this up to ghci. https://github.com/tonyday567/chart-svg
Compl Yue
@complyue
@tonyday567 the experiment with ghc is minimalism, the purpose is to delegate the initialization of GHC api state to stack ghci, so easily the ecosystem of haskellstack is right available to hadui. that said, I don't think data analysts could be afraid to use a custom compiled GHC (with SIMD vectorization optimized e.g.) after all, that's one of the greatest things with open sourced tool stack.
Stefan Dresselhaus
@Drezil
@complyue depends on how you can set things up. If you just have a "download, unzip & run"-solution there would be no friction of adoption. If people need to clone, patch & compile themselves then i don't think that will be a way to go..
Compl Yue
@complyue
@Drezil can't agree more, ultimately hadui should just work out-of-the-box with stack build --exec hadui in any new stack project. but I'm not hurry toward that before it's prepared for wider audience. the most critical issue is the arbitrary-code-execution vulnerability for now, and I don't think it's realism to have a GHC minor release on 8.6 branch even I submit a MR. GHC 8.8 is still out of stack lts right now, I'd think of a MR to 8.8 branch once stack lts embraces it.
Compl Yue
@complyue
stack itself has not matured the custom GHC instance feature enough, or I could use that to package a custom GHC package for easy get. all need just time I'd think ;)
Stefan Dresselhaus
@Drezil
why not put a MR into ghc 8.8 before stack embraces it?
for the latter one: hvr has his GHC-ppa .. so it should be easily doable for the common ubuntu-folks.. https://launchpad.net/~hvr/%2Barchive/ubuntu/ghc
Compl Yue
@complyue
um, I'd like to stick with stack, for all my team to use just stable features of it. and I'm prioritizing works at hand acroding to internal needs so so.
Stefan Dresselhaus
@Drezil
ah, ok :)
Compl Yue
@complyue
btw, I've haskelled not long, does not many ones compiling GHC for themselves? when I think of it, what's in my mind is tensorflow, there you almost always compile it yourself as the stock release is way to conservative about hardware requirements, you have to compile with sse4.2 avx etc enabled to not waste capabilities on decent CPUs
Stefan Dresselhaus
@Drezil
not really .. as with tensorflow you don't care about ssse4 & avx if you use a graphics-card anyway ...
most is: pip install tensorflow & use it.
at least in my experience..
might be different if you have a big, dedicated team at a big corporation .. but most of my experiences are in science (students to post-docs that are happy when it just works(tm)) and in small businesses where you just have 1-2 people doing ML
Compl Yue
@complyue
okay, my team is less than 10p, but have to spin up dozens of rack servers at constant crunching, so that's not typical :)
Guillaume Desforges
@GuillaumeDesforges
Hey everybody!
I was thinking about writing some sort of cached pipeline for my Machine Learning experimentation workflow. For the mechanism I am thinking about, I need to compare the running function plus inputs with previously ran functions and inputs. I was thinking about comparing hashes.
Would it be possible to get the hash of a piece of source code (at compile time I guess)?
Yves Parès
@YPares
@GuillaumeDesforges Hi! Hashing the source code would be difficult to do without some template haskell trickery. http://hackage.haskell.org/package/funflow does what you want, and it just requires you to update a salt whenever you modify your code and want to invalidate your cache
Tony Day
@tonyday567
@complyue Is hadui like a ghci replacement, similar to intero, say, with interactive custom commands, or is it more fundamentally a different way to use GHC? Or something else?
Compl Yue
@complyue
@tonyday567 my own usecase is to have vscode+hie open developing all code of the stack project, with hadui-dev run as the ide's build task (it keeps running all the time, when runtime errors occur, code), then a browser window open hadui page to enter interactive code, mostly parameter scripts to trigger plotting in the web browser.
Compl Yue
@complyue
hadui-dev will print source location to the ide console when runtime error occurs, so you just click the link to navi to source in vscode. i'm just now working on the error formatting code, and found non-trival to extract useful info from runtime errors with ghc :/
Compl Yue
@complyue
planned hadui-dap to support breakpoint, single stepping, var reveal etc in vscode to debug hadui project, while no tight schedule for that.
Compl Yue
@complyue
Surprise! it may be painless for you to try out hadui right now. I was wrong to recognize stack's custom GHC instance feature as immature, that coz I'd built GHC with its new hadrian method, I'm amazed to find out that stack actually works pretty well with the bindist of GHC built with good old make.
so you are 3 cmds away from a running hadui-demo on your machine, given you are on decent macOS or Linux. please let me know whether it succeeded or not if you ever tried. https://github.com/complyue/hadui-demo
Marco Z
@ocramz
@complyue trying the demo (osx mavericks), but my system doesn't have xz it seems
this project is a very cool idea, my only concern (pretty much like @tonyday567 ) is with the custom ghc build. Congratulations for getting it all to work together though!