Say what. If we can give people good input and output based on solid types, everyone will be incentivized to write to these types.
So, for example, if we can send a CSV file through type X into graphing in one line, people will happily work with X because they will start from this.
I agree! Too often Haskell libraries are clearly designed to prove some generality (valuable academic work) rather than designed to get some job done. Start with some key tasks a person wants to accomplish and build the library around that. Types can follow the purpose
Maybe we can start with the abstraction of a data set. This can be a directory of CSV files or an SQL data base. What is the appropriate type? Not an easy question already.
What is the simplest thing that can be done with a data set that is still useful?
I immediately think of https://datasette.io/
To glue two data sets together is one simple thing I can think of. Like, a canonical data set monoid.
There is also visualization. In principle every data set is isomorphous to its visualization and its storage.
1 reply
There are also projections — destructive operations like an average of a column or a correlation between columns.
Finally, there are generative operations that create «fake» data sets from a bunch of parameters. They may be either various noises or carefully derived predictions.
There is also distance between data sets of the same type that lets us do validation.
Suppose we split the data set into 10 random slices. Do we now have a data set of data sets?
Now we can talk about how well a given prediction is validated as a projection of the data set of data sets. For example, we can visualize it. Cool!
Yes, what I mean is not a single visualization. There is a «full set» of visualizations that displays all dimensions of all data points.
Something like a scatter plot matrix.
We want to be able to see all the information that there is, so at least there is a monic arrow from the type of data sets to the type of visualizations.
And a visualization can be written down as a data set — we want it to be the same data set we started with.
Maybe we can say that there are many different visualizations for the same data set. Like, say, with different colour schemes.
But, for example, a box plot is not a visualization. It is visualization after projection.
Interesting! I see where you're going.
But is that actually useful? :) Datasets don't just exist to be manipulated, but to explain something useful about the real world. What I mean is that a tool to simplify creating a box plot out of a CSV file is more likely to build a user base than a general algebra over datasets
I can easily build a tool that does something specific for Joe, Kim and Mary. But no amount of such tools will get Haskell to a place I want to put it in. We must show developers that they can develop, and then developers will show users that they can use.
Bold claim: every widespread language has at least one widespread framework, in the wide sense of the word.
- Python: Flask, PyTorch.
- JavaScript: DOM API, React.
- C: Linux, the POSIX standard.
- R: the R repl.
I say success of a language is impossible unless hard decisions are already made for some practical area of expertise.
This does not imply causality of course. It is hard to establish causality in History. So, I find this observation persuasive — this is as good as it gets. If we are to make any decision, the decision to get a framework going is a good one.
Haskell's "framework", so far, is geared towards language design
1 reply
I stumbled upon dataHaskell recently and thought it looked cool. Then I decided to give it a try, first by using Chart. I spent the entire day trying to set it up in a stack project but failed because there was no LTS version providing all the necessary component libraries of Chart(?). Then I found out that the Chart repository has basically been inactive since 2016 or so...
This community seems largely inactive especially recently but still, I love both Haskell and data science and I think it'd be really cool to have a Haskell environment where you can do advanced data science stuff easily. If I wanna contribute to dataHaskell, where can I start?
Seems so. I guess it would help to start by clarifying the ideas. I mean, what would really help is if some hacker genius just threw together a new tool and language that was heavily influenced by Haskell on the one hand, and successful tools on the other. Maybe it would be an edsl, maybe not
hi tebu1783 (tebu1783), I second your sentiments re. the need for this kind of tooling in haskell, and of a supportive community
but this language is both a blessing and a curse because it favors experimentation and is very large, therefore there are more "standards" than actual users
we also have to reinvent many things from scratch that other languages get for free like subtyping or inheritance
@tebu1783: if there's not suitable LTS, would cabal work?
I'm a Nix advocate, I'll try to keep down the zealotry to a minimum though
Thanks for the response, @ocramz_:matrix.org
That reminded me of the classical essay The Lisp Curse. I thought this wouldn't be the case for Haskell because it's way more restricted than Lisp (in the sense that it has no macros)... But the "Haskell Curse" is still caused by the enormous amount of language extensions and new advanced features built on top of those?
That reminded me of the classical essay The Lisp Curse. I thought this wouldn't be the case for Haskell because it's way more restricted than Lisp (in the sense that it has no macros)... But the "Haskell Curse" is still caused by the enormous amount of language extensions and new advanced features built on top of those?
@man_of_letters:mozilla.org @b:chreekat.net I haven't given cabal a go, but now you mention it, I should try using Nix and cabal as a last resort, too :-)
Fwiw I don't feel like the number of language extensions is a problem, especially with the GHC20xx concept that is now available. But the fragmentation of the "standard" library is a problem
And the other problem as far as data goes is the Num type class
see? that's exactly what I meant above: 2 users, 2 orthogonal opinions.
IMO the haskell type system should simply be bypassed for data/ML workflows. With metaprogramming, generics and so on. Stuff like type families and singletons is simply a pain in the ass at scale.
this is my rationale for 'heidi' and my recent work on tensor compilation
Yeah I'm thinking the same thing, in that the boilerplate of type decorations is just a tremendous pain in the ass sometimes. And I agree that opening the doors to over engineering is probably not the right idea in this situation
My personal experience in this domain was at university studying numerical analysis and computational physics. For some dumb reason I really wanted to use C for everything at first. When I got around to anything using matrices, though, the boilerplate just killed me. I switched to octave. Sure, having a bunch of math stuff built in was useful, but what really saved me was a compact syntax built for the problem domain.
That's why I'm not sure just using raw Haskell would be the right idea.
I never got around to learning python, so I'm curious to know in what ways it was flexible enough to get heavy adoption for mathematical programming
chreekat: python is great for doing science because it runs your code no matter what. No matter what they say about pre-registering experiments, scientists just love to tinker and make up hypotheses after the fact
haskell is the opposite, it forces you to formalize the data domain very early. And the more type stuff you add to the domain, the harder it becomes to refactor