+1 hvega from me too. vega-lite itself has been such a boon to other languages, thanks for your work on hvega @DougBurke.
@YPares did you ever find a sql solution? i'm using sqlite-simple a lot... i can live with sqlite-simple but i think i have to bite the bullet on some minimal string builder on my current project. it'd be nice to have more lightweight string-builder middle-ground options between hard-coding queries as strings and full-on orm commitment.
@frasermince can ping us in the hasktorch slack if you'd like to get a hasktorch kaggle effort going. will say kaggle is probably a space where haskell has few comparative advantages (there's no penalty to scripts being hacky/throwaway as long as they score the test data, this is not true of "real-world" ml).
Hi all. I'm working as a data engineer for a startup and I'm trying to push forward functional programming for our data processing. I've read an interesting discussion when looking at Scala capabilities. They say:
I think the main thing holding us back is a good data frame implementation that the community settles on.
What's the current state of that in Haskell? If I google "haskell dataframe" I stumble upon the Frames lib. Is it the "community standard" lib to do data processing?
heidi works and is "usable" (albeit a bit rough around the edges) but I got stuck on how to pretty-print dataframes (ocramz/heidi#2) and how to reconcile tries with lists as keys and relational joins (ocramz/heidi#13 ). The first one is a somewhat mind-bending puzzle involving generics, and the second one more of an UX problem. Having other eyes on this would be a great help!
A question: is there a way to move directly to a useful Heidi row without declaring an instance for a record? What I really want, what is often hard in Frames, is to add columns on the fly. For example, recently I have been using Stan to model things and then produce results with confidence intervals. So I get, e.g., a
Map, keyed by one kind of data, with values as lists of doubles representing 5%, 50%, 95% or whatever. I’d like to make those into Heidi rows, add some cols, merge, via stacking and/or joining, with other rows, maybe compute an average or whatever, then feed all that to hvega for charting. Nowhere in there do I really need a corresponding record, just named and typed columns which I can efficiently access, join, concatenate and fold over, then extract by name and type. Does that make sense?
Vinyl.Record rs -> Heidi.?? function what is ???
at can be used both for lookup and insertion : https://hackage.haskell.org/package/heidi-0.0.0/docs/Heidi.html#v:at
Map Text [Double] and I want Heidi.Frame (Row [TC] VP) (I think). I can sort of see how to do it with rowFromList except toVal is not exposed so I can’t make my values into VP’s. What’s the canonical way to encode directly?
let toHeidiRows :: Map Text [Double] -> Heidi.Frame (Heidi.Row [Heidi.TC] Heidi.VP)
toHeidiRows m = Heidi.rowFromList $ fmap f $ M.toList m where
f (s, [lo, mid, hi]) = [(Heidi.mkTyN "State", Heidi.toVal s)
, (Heidi.mkTyN "lo", Heidi.toVal lo)
, (Heidi.mkTyN "mid", Heidi.toVal mid)
, (Heidi.mkTyN "hi", Heidi.toVal hi)
]VP works for that one case, but I don’t think that’s a good way forward. What I want is something like Vinyl.AllConstrained Heidi.Heidi rs => Vinyl.Record rs -> Heidi.Row [Heidi.TC] Heidi.VP or something like that. But the path from toVal at the element level, to a row is not clear to me.
[TC] part? I’d like to put the column names from the Record in there but I think you want the last element of the list to represent the type of the column, right? So add a TC to that list for the column name or is there some better way to do this?
rfoldMap https://hackage.haskell.org/package/vinyl-0.13.1/docs/Data-Vinyl-Core.html#v:rfoldMap is the right combinator
@unhammer fwiw transformer models are on the critical path for a project i'm working on. can't speak for torsten but i think it's pretty important for the things he's interested too. it's also a space of capabilities that i think are hard to accomplish any other way in haskell so i think it's important to the ecosystem from that perspective.
bottom line is, these factors are going to move things effort forward in the medium term and we're definitely looking for help from others that want to see this happen.
The implementation path is pretty clear, there's 3 main things that need to get done:
1) mature the haskell bindings to rust tokenizers https://github.com/hasktorch/tokenizers
2) for cases where training is desired - will want to implement the haskell architecture and a weight parser that's 1:1 to deserialize the pytorch version. For cases where you just want to do inference and use haskell to deploy a model the usual torchscript inference + tokenizers will probably be sufficient.
3) address library performance pain points along the way (for example - mutable optimizer patterns for training w/ large # of parameters).
