(There should be a word like "large" defined to mean DoesNotComfortablyFitInRAM)
:+1:
Looks like
compact might not be a fit; I have one binary that does the processing-dumping and other binaries that do the reading. Maybe I'll just roll serialize instances for all the data . . .
I recently added a tutorial to hvega (an API to create Vega-Lite visualizations) - based on the Elm tutorial - and would be interested on feedback. You can find it on hackage at http://hackage.haskell.org/package/hvega-0.4.0.0/docs/Graphics-Vega-Tutorials-VegaLite.html
@o1lo01ol1o @DougBurke I’ve worked at that some, esp using Frames to add some types to the data. I’ll post a github link later. But once you get used to it, most charts are pretty sui generis and the templates for them ended up being as limiting as they were useful. The examples are the most useful thing, I think.
@Shimuuar - an aim of Vega Lite (I believe) is that you should be able to "compactly" specify a visualization without having to define everything. That is, it is rather declarative - "plot up this column versus this column" and it will decide on axis ranges, titles, symbols etc. You can then add in extra "options" to change the default choices. Now, whether hvega manages to capture "compactly specify" is a different question. I would appreciate any comments on the existing documentation and how they could be improved.
I should add that as well as the frames interface by @adamConnerSax there is also http://hackage.haskell.org/package/rigel-viz by @ocramz (which I haven't looked at)
I only found out about rigel-viz when I was looking for a category for http://hackage.haskell.org/package/vega-view - which is a possibly-pointless web server to let you view on-disk vega/vegalite visualizations (aka I could have just installed https://github.com/vega/vega-desktop but decided to write omething in Haskell instead)
@DougBurke I finally managed to plot a parabola using hvega. I wouldn't call it compact:
let enc = encoding
. position X [ PName "X", PmType Quantitative ]
. position Y [ PName "Y", PmType Quantitative ]
let dat = dataFromColumns []
. dataColumn "X" (Numbers [1 .. 10])
. dataColumn "Y" (Numbers $ map (^2) [1 .. 10])
toVegaLite
[ dat []
, mark Line []
, enc []
]I'd say it's completely unsuitable for interactive plotting. Why should I specify name of column both in data and axis specification? Surely it could be inferred from data type [(Double,Double)]
So you can then wrap that how you like with much more inference. I do some of that with Frames, etc.
But it requires some use, for me anyway, to see what I would like inferred and what not. For publication ready charts you often want to fiddle with the details and even just for data exploration I’ve found that I rarely use much simplification except the Frames part. Perhaps a different question is what would a truly “haskelly” grammar-of-graphics API look like? Maybe we could work towards that and then use hvega (and other things!) as back-ends.
But some of that arbitrariness that comes from the “stringiness” of the column names can def be handled using a typed data frame (like Frames) and doing some wrapping. YMMV, of course.
I suspect I'll end up doing something along these line. One thing is designing good APIs is difficult. Especially ones intended for interactive use
Generally I think one shouldn't need do much beyond plot someData on one hand. On other hand sometimes plots will be tweaked to death and this shouldn't be difficult either
@Shimuuar On another topic, I've been using
monoid-statistics and find myself needing windowed statistics. There exists some mentions of monoidal-rolling statistics , ekmett has a semi-cryptic tweet here which unfortunately isn't enough to derive an implementation from (that I can see anyway) and this paper that also uses an okasaki queue and some pointer swapping that seems less pleasant to implement. Have you looked at any approaches?
I'll think more about it. Seems useful
So I keep hoping kmett will pop up in IRC so that I can pester him about it.
@Shimuuar as @adamConnerSax says, the point of hvega is to provide some way to create a Vega-Lite specification. It is based on the Elm API because that required little thought to implement. I would love there to be a "higher level" API, but whether there really will be one API to rule them all, or a range of them, is to be seen. I don't think I am the person best placed to design such an API, but we already have some examples (with http://hackage.haskell.org/package/rigel-viz and https://github.com/adamConnerSax/Frames-hvega).
Some extra complications for the vega/vega-lite design space at this time: the underlying JSON specifications are moving pretty fast, so while I would appreciate a more type-safe API than hvega provides, it is not obvious to me how to do this and be responsive to changes in the specs.
@DougBurke @Shimuuar I’m pretty happy with hvega as is but I would also love a more strongly-typed interface. I’ve thought some about that but I’m not quite sure I have a clear handle on that API design. Partly, I am starting to see how useful the g-o-g ideas are but don’t yet understand it well enough to see what those ideas look like mathematically. I like that hvega is tracking the vega-lite spec and I think a higher-level g-o-g API would be great, esp if it had an hvega and diagrams back-end.
@DougBurke Anyway, thanks so much for the work! You can see some in action here: https://blueripple.github.io/research/preference-model/p1/main.html
(I should say that Jo Wood, who created the Elm version that hvega is based on, really deserves the credit, I'm just along for the ride!)
Somewhat unrelated. Plotting APIs tend to create tons of names. On one hand you don't want to dump them into unqualified namespace on the other haskell (and ML-family) syntax is outright hostile to qualified names:
Plot.axes . Plot.coordX . Plot.label .~ "Cool label" I wish for something like open Plot
All names from Plot could be used unqualified here
I am thinking of setting up an azure vm with cuda - any views on what would be a good entry level machine? https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-gpu
I set up an account about 6 years ago but only started really using it about 6 months ago
You can just use an ubuntu machine on paperspace and install nix
"just"