So I would tag Julia as a bit player just like us. I don't know too much about the language, but I read that it is the direct opposite to Haskell in many ways when it comes to typing etc. Which makes it a very useful comparator. Anything the reduces the empire of python is a good thing when it comes to alternative approaches becoming more acceptable.
Julia is a more affected by lisp than Haskell in language design, and it is dynamically typed - you can start writing in Julia without mentioning types. The package manager there is very friendly too, so is community - people like sharing when they write something in a new language.
Hi
@rohakapo sorry, I'll turn on the notifications.
I'm very beginner at haskell and also I'm very beginner at DataScience.
But I love haskell and this language give me more satisfaction than anything in IT.
I'm looking to make a carer in data science. I'm from python backend word.
Any advice?
Chart's API is broken enough but plots themselves are decent. I don't like anything else so I just forked Chart https://github.com/Shimuuar/Chart-B/tree/ChartB (branch Chart-B). It's very much UUU (unfinished, undocumented, unstable)
1 reply
@idontgetoutmuch I’ve been using hvega https://hackage.haskell.org/package/hvega (my target is producing html). Also some boilerplate but I like the grammar-of-graphics way of structuring things.
Suggestions for improvements to the
hvega docs - in particular for the "how do I actually display the damn thing" - are appreciated at https://github.com/DougBurke/hvega . I'd particularly like to improve the IHaskell story, but that (I believe) requires IHaskell work which I haven't had the energy to dig into.
If you create JSON output - I see I don't really document how to do this but
spec = A.encodeToLazyText (fromVL vl) should do - then save it to a file you can then use https://github.com/DougBurke/vega-view (or the vega desktop application) to view it (it runs a web server that will let you view the .vg.json files in the working directory as well as allowing you to drag-n-drop files onto the browser).
+1 hvega from me too. vega-lite itself has been such a boon to other languages, thanks for your work on hvega @DougBurke.
@YPares did you ever find a sql solution? i'm using sqlite-simple a lot... i can live with sqlite-simple but i think i have to bite the bullet on some minimal string builder on my current project. it'd be nice to have more lightweight string-builder middle-ground options between hard-coding queries as strings and full-on orm commitment.
@frasermince can ping us in the hasktorch slack if you'd like to get a hasktorch kaggle effort going. will say kaggle is probably a space where haskell has few comparative advantages (there's no penalty to scripts being hacky/throwaway as long as they score the test data, this is not true of "real-world" ml).
Hi all. I'm working as a data engineer for a startup and I'm trying to push forward functional programming for our data processing. I've read an interesting discussion when looking at Scala capabilities. They say:
I think the main thing holding us back is a good data frame implementation that the community settles on.
What's the current state of that in Haskell? If I google "haskell dataframe" I stumble upon the Frames lib. Is it the "community standard" lib to do data processing?
@GuillaumeDesforges I’m not sure there is a “standard.” Frames is what I use most of the time. But sometimes Haskell’s limitations in constraint manipulation—though maybe this could be improved by a plugin or using newer type-level features in ghc—has me reaching for a representation more akin to Heidi (https://hackage.haskell.org/package/heidi), that is, something less precisely typed. Frames is great when you are writing functions to manipulate a known set of columns. Trying to write polymorphic Vinyl/Frames code can be tricky because you have to prove a lot to the compiler to convince it that the manipulation you are doing is correctly typed. In a sense, this is the point! Catch a whole bunch of errors at compile time! But it’s sometimes harder than it seems like it ought to be. Of course, YMMV.
My feeling as a newcomer is that few people use either because at least Frames is not trivial to use (the fact that there are only few users => lack of examples/support). As an ex-pandas user and average haskell programmer I find Frames great, well-designed etc but I miss some features etc. In my experience the compiler errors can be completely misleading. I guess a ghc plugin (like for polysemy) could make the experience a lot better but I dont know enough to be sure.
Speaking of which,
heidi works and is "usable" (albeit a bit rough around the edges) but I got stuck on how to pretty-print dataframes (ocramz/heidi#2) and how to reconcile tries with lists as keys and relational joins (ocramz/heidi#13 ). The first one is a somewhat mind-bending puzzle involving generics, and the second one more of an UX problem. Having other eyes on this would be a great help!
my "give" to the community instead is this: https://hackage.haskell.org/package/splitmix-distributions-0.7.0.0/docs/System-Random-SplitMix-Distributions.html a new package with various (univariate) random samplers, both continuous and discrete.
@ocramz Interesting! I’ll look at the issues. I once did a bunch of stuff with generics-sop but as I recall, it takes some time to get your head around…
A question: is there a way to move directly to a useful Heidi row without declaring an instance for a record? What I really want, what is often hard in Frames, is to add columns on the fly. For example, recently I have been using Stan to model things and then produce results with confidence intervals. So I get, e.g., a
A question: is there a way to move directly to a useful Heidi row without declaring an instance for a record? What I really want, what is often hard in Frames, is to add columns on the fly. For example, recently I have been using Stan to model things and then produce results with confidence intervals. So I get, e.g., a
Map, keyed by one kind of data, with values as lists of doubles representing 5%, 50%, 95% or whatever. I’d like to make those into Heidi rows, add some cols, merge, via stacking and/or joining, with other rows, maybe compute an average or whatever, then feed all that to hvega for charting. Nowhere in there do I really need a corresponding record, just named and typed columns which I can efficiently access, join, concatenate and fold over, then extract by name and type. Does that make sense?
but you certainly should be able to work with the Row types directly
@ocramz I sort of figured but then I couldn’t figure out the best way in. Do you have an example of that sort of use, or a suggestion for where in the Heidi code to look? Or, more simply, what is the type of a “row” in Heidi? E.g., in a
Vinyl.Record rs -> Heidi.?? function what is ???
@adamConnerSax the interface to Row is here : https://hackage.haskell.org/package/heidi-0.0.0/docs/Heidi.html#g:13
it's pretty much Map-like, and there are some lenses for extracting values at the right type : https://hackage.haskell.org/package/heidi-0.0.0/docs/Heidi.html#v:int
more lenses , for example
at can be used both for lookup and insertion : https://hackage.haskell.org/package/heidi-0.0.0/docs/Heidi.html#v:at
@ocramz Cool. How do you suggest I handle the typing bit when building rows? I can see how the generics works but I want to do that bit by hand. E.g., I have a
Map Text [Double] and I want Heidi.Frame (Row [TC] VP) (I think). I can sort of see how to do it with rowFromList except toVal is not exposed so I can’t make my values into VP’s. What’s the canonical way to encode directly?
Sorry, more like:
let toHeidiRows :: Map Text [Double] -> Heidi.Frame (Heidi.Row [Heidi.TC] Heidi.VP)
toHeidiRows m = Heidi.rowFromList $ fmap f $ M.toList m where
f (s, [lo, mid, hi]) = [(Heidi.mkTyN "State", Heidi.toVal s)
, (Heidi.mkTyN "lo", Heidi.toVal lo)
, (Heidi.mkTyN "mid", Heidi.toVal mid)
, (Heidi.mkTyN "hi", Heidi.toVal hi)
]
@ocramz ^
@ocramz So, exposing the constructors of
VP works for that one case, but I don’t think that’s a good way forward. What I want is something like Vinyl.AllConstrained Heidi.Heidi rs => Vinyl.Record rs -> Heidi.Row [Heidi.TC] Heidi.VP or something like that. But the path from toVal at the element level, to a row is not clear to me.
Val is just an internal type to deal with recursive values, but it all gets flattened down to a single layer
That makes sense. But I’m not sure what is best in the
[TC] part? I’d like to put the column names from the Record in there but I think you want the last element of the list to represent the type of the column, right? So add a TC to that list for the column name or is there some better way to do this?
but if starting from a vinyl Record, I think
rfoldMap https://hackage.haskell.org/package/vinyl-0.13.1/docs/Data-Vinyl-Core.html#v:rfoldMap is the right combinator
where each output key would be a list with a single TC inside