For questions, use https://discourse.holoviz.org/ instead! This chat is for informal discussions and debugging sessions with developers of the HoloViz.org tools (HoloViews, Datashader, Panel, hvPlot, GeoViews, Param, etc.), but user questions should go to the HoloViz Discourse site so that others won't have to ask them again.
Thanks @jbednar , problem solved - this actually works, the backend has to be specified at the panel level, and there is a warning - I just missed in among all the warnings the system shows - it tells me it does not recognize backend by Holoviews - This is good. If I move fast enough I will post the tutorial skeleton on Github under CC0 by tonight.
I will go on and figure this out when everything is ready - it worked before so its a minor issue that can be taken care of.
Hi everyone, does hvplot support heatmap with datashading? Could not find an example in the docs and adding
datashade=True to the examples in here causes an error. About to file github issue but just wanna double check
Usually you'll add datashade=True on whatever your original data was, not on .heatmap.
So e.g. .hvplot.scatter(... datashade=True), etc.
Once it's already a heatmap there's not much left for Datashader to do, as basically what Datashader does is to make heatmaps!
Ah, so if my data looks like this, and I want to visualize this giant heatmap
then I just do .heatmap(columns="columns", index="index") right?
I ask bc when zooming in it is slow, so I thought it was somehow not optimized yet
maybe I just need to connect to a more powerful dask backend and turn that matrix to a
dask dataframe then call the heatmap function again?
You could plot that with Datashader by creating an image out of it and datashading that (not sure how in .hvplot, but in HoloViews it's hv.Image(...)), but that requires you to throw away your categorical axes and replace them with numerical ones. Datashader only knows about numerical axes. If there is some reasonable ordering, then maybe that makes sense. If there's no ordering, there's no point in such a viz, as Datashader would then be aggregating across adjacent columns and rows that have nothing in common.
Whether it's Dask or Pandas or Numpy doesn't matter; whether it's categorical data does matter. If you can map from categorical data into numeric data, then Datashader can handle it. If not, it has no idea what to do with your categorical x and y axes; it's built for continuous variables on x and y (though it does support categories on z, i.e. stacked up together).
Zooming in won't really help if the data has no structure; if it's truly categorical in x and y you just have to deal with the data in chunks somehow, because categorical data isn't meaningful without a label.
It would just be meaninglessly grouping unrelated values.
Datashader can handle when the data has spatial structure (in whatever space you are plotting things in). Without spatial structure, zooming is not meaningful.
But then you'll want to reveal categories when you've zoomed in to individual data points, which I guess you can do with hover.
It would be great to add something to the examples where we show that whole process, i.e. starting with categorical axes, clustering the data to give spatial structure, then displaying the datashaded result with some way of indicating underlying categories (with colors plus hover, etc.) That would make a nice example, but it's a good bit of work, with lots of assumptions along the way. In no way would that get packaged up as
.hvplot.heatmap(... , datashade=True). :-)
It's somewhat similar to https://examples.pyviz.org/iex_trading/IEX_stocks.html, where zooming in reveals individual stock trades, and zooming out reveals patterns; you'd need some similar approach, plus the clustering.
thank you @jbednar for the rapid help :)
1K x 1K isn't so bad, actually, given current monitor resolutions, because at least then all the data will fit on screen. The problem is then just that the data means nothing without the labels. So If you do only want something where there's no actual aggregation (Datashader's specialty), i.e. no downsampling required, then I withdraw my objections about it being meaningless without clustering. Clustering would help a lot, but if every category bin at least gets one pixel, it's not ridiculous to simply datashade the array.
In that case it's a simpler problem to solve, but not a solved one, because we don't support datashade=True on heatmap directly, which means that we can't provide the categorical axes along with the datashaded plot. And we'd have to do a good bit of work on Bokeh to subsample the category values so that it doesn't try to draw 1000 labels. But that's doable, if time consuming.
Definitely not something for anyone but a hardcore Bokeh hacker to attempt, though!
(My objections above are about when there are more rows and columns than in your plot, but that doesn't have to be true if you only have 1K x 1K. At 3K x 3K it's already a problem again, of course...)
just to follow up, assuming that it's 1k x 1k, you mean to do like what they did in here right? http://holoviews.org/user_guide/Large_Data.html (Hover info section)
Using rasterize without shade
And if we see hover info from a datashader output, it means that it's not aggregated right? and if we cannot hover over then the data has been aggregated. Is that correct?
In any case, no, I don't think that support would work here. The hover in this case will either only show the row and column and value of the datashaded plot (which isn't useful, since it won't have the category axis values), or it would require overlaying something as large as the original table (which will then incur the same slowdowns you've been seeing).
What's needed is a hybrid, where Datashader renders the heatmap, Bokeh knows the list of categories in each axis (but not the full crossproduct between those two axes), and Bokeh displays a dynamic subsampling of the category values depending on the zoom level. So it would be a datashaded plot, with magic axes.