Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 17:49
    MarcSkovMadsen edited #2408
  • 17:49
    MarcSkovMadsen labeled #2408
  • 17:48
    MarcSkovMadsen milestoned #2408
  • 17:48
    MarcSkovMadsen commented #2408
  • 17:48
    MarcSkovMadsen commented #2408
  • 16:13
    philippjfr commented #2406
  • 16:12
    mathrick commented #2406
  • 15:34
    philippjfr commented #2408
  • 09:03
    philippjfr commented #2403
  • Jun 19 19:44
    MarcSkovMadsen labeled #2388
  • Jun 19 19:43
    MarcSkovMadsen commented #2388
  • Jun 19 19:43
    MarcSkovMadsen commented #2388
  • Jun 19 19:39
    MarcSkovMadsen milestoned #2389
  • Jun 19 19:39
    MarcSkovMadsen labeled #2389
  • Jun 19 19:29
    MarcSkovMadsen milestoned #2394
  • Jun 19 19:29
    MarcSkovMadsen labeled #2394
  • Jun 19 19:29
    MarcSkovMadsen labeled #2394
  • Jun 19 19:29
    MarcSkovMadsen commented #2394
  • Jun 19 19:28
    MarcSkovMadsen commented #2394
  • Jun 19 19:26
    MarcSkovMadsen milestoned #2395
James A. Bednar
@jbednar
Yes, click on the links at the top of the page to see the live version. Plus you have to zoom in enough that a sufficiently small amount of data is required.
At that point, that section of the data is no longer being datashaded, it's just a normal Bokeh plot, with hover, selection, etc.
The dynamic switching is what would be of interest here.
hoangthienan95
@hoangthienan95

Awesome! Gotcha. Thank you for patiently explaining it to me, I'm new to this stuff.

I'll try that first, then if it doesn't work for some reason I might explore other tools. My only other two that I know for large data viz for python is Plotly with WebGL rendering and then Vaex with interactive widget . Have you had experience with either? Do you think any other libraries would be promising

I'll attempt at the discourse post and will probably bug you again to make sure I mean the right thing

James A. Bednar
@jbednar
Bokeh also has webgl, but I don't know if it works with heatmaps. https://higlass.io/app may be of interest. Higlass is a good example of dynamic categorical axes; I just wish they had used Datashader behind the scenes so that it's compatible rather than competitive!
I don't know of any interactive categorical axes with Vaex, but if there are, I'd be interested to see them.
hoangthienan95
@hoangthienan95
The stuff in higlass page is exactly the kind of data I'm working with every day haha. Classic Biologists, loves to reinvent the wheel (sometimes that's good but sometimes bad)
James A. Bednar
@jbednar
If HiGlass works, just enjoy! :-)
But what I want is what's there, but with no tie to Biology or any other specific domain; there was no need for it to be so domain specific. Sigh!
image.png
Are these numbers being updated. To me they have looked the same for 6 months. Maybe my memory is playing tricks on me.
image.png
hoangthienan95
@hoangthienan95
Thank you @MarcSkovMadsen
epifanio
@epifanio
I was wondering ig the GPU processing is used automatically by datashader, when a cudf/cupy environment is found
Philipp Rudiger
@philippjfr
Sorry, meant to reply but didn't get around to it.
Quick reply: "Sort of", i.e. it's automatic when the input is on the GPU, e.g. i.e. if you're feeding in a cuDF
epifanio
@epifanio

@philippjfr thanks!
I am porting my little app from bokeh to panel https://github.com/SIOS-Svalbard/NC-Plot
( bdw, thanks to the awesome presentation at pydata berlin 2019)

in my app I use xarray to read a netcdf datasource, and then transform it to a pandas dataframe which is then feeded to a bokeh columndatasource

I will first complete the porting into panel (from bokeh) then I will add a check to understand if cudf can be imported and if yes do a further transformation from pandas dataframe to cudf and give it to datashader - i guess it is not possible to go from xarray directly to cudf

Philipp Rudiger
@philippjfr
What kind of data is it?
epifanio
@epifanio
3 type of data, all stored as netcdf: Time Series, Profiles, TimeSeries-Profile - the app takes an url as input and has to guess the data-type
based on the type, a different widget/app is composed - there are 3 demos, the more interesrting (embarassing simple) is the time-series-profile
epifanio
@epifanio
they are CTD observation, weather station, soon permafrost - but we also have to integrate gridded data (sentinel satellite)
Philipp Rudiger
@philippjfr
Okay for gridded data you could also convert to cupy (I think xarray supports those now) and then use a HoloViews QuadMesh + datashading to render it.
Would recommend https://hvplot.holoviz.org/ for plotting, at least in the exploratory phase.
I don't know how large any of your data is so not sure how much benefit you are going to see from datashading or if you even need GPU datashading.
But hvPlot/HoloViews will transparently work with either cuDF or regular dataframes and xarrays backed by NumPy, Dask or Cupy.
epifanio
@epifanio
For gridded data I started with the landsat example which is using the xarray_raster capabilities, so I guess I will first use xarray_raster to read the data in and then convert it to a cupy array which should be understood by the quadmesh api
each nectdf data (for the gridded data) is ~2GB
James A. Bednar
@jbednar
Sounds good.
Thomas Diederen
@Patrickens_gitlab
is it possible to label datapoints in hv.Scatter? In the following example I would like a legend where the datapoints are labelled 'a' and 'b' (corresponding to color):
import holoviews as hv
import pandas as pd
import numpy as np
import random

randhex = lambda : format(random.randint(0, 255), '02x')
randcol = lambda : f'#{randhex()}{randhex()}{randhex()}'
data = pd.DataFrame({
    'x': np.arange(0,1,0.1),
    'y': np.arange(0,1,0.1),
    'color': [randcol()]*5 + [randcol()]*5,
    'label': ['a']*5 + ['b']*5,
})
hv.Scatter(data, kdims=['x', 'y'], vdims=['color', 'label'], label='label').opts(color='color', show_legend=True)
Thomas Diederen
@Patrickens_gitlab
but thats for labels in the plot, I am looking for 2 legend entries: 'a' and 'b' corresponding to the colors
and thank you for the answer!
James A. Bednar
@jbednar
You just want to turn on a legend? I forget the name, but there's an option for that!
Thomas Diederen
@Patrickens_gitlab
I looked through help(opts.Scatter) but could not find an option that I would think could do that
Thomas Diederen
@Patrickens_gitlab
should I think about something like: categorical_points.sort('Category').options(color='Category', cmap='Category20', size=5)? This is straight from the documentation, but I cannot get it to play nicely with my dataframe
James A. Bednar
@jbednar
Ah, I wasn't paying close attention to what you are doing. A legend will be generated for an overlay of points on points, to distinguish between two different sets of datapoints. So if you separate your points into two different Scatter elements then overlay them, you'll get the legend you want. You can use a groupby to set up such categories. But if you have a column with your color, you could supply an infinite number of different 'label' arguments, which isn't something legends are set up to handle, so there's no direct way to set up a legend for that case that I know of. So, first get a finite set of different groups, then plot those as an overlay.
Thomas Diederen
@Patrickens_gitlab
ok, ill post a solution in a bit to see whether Ive understood what you mean. Thank you!

as such:

randcol = lambda : f'#{randhex()}{randhex()}{randhex()}'

data = pd.DataFrame({
    'x': np.arange(0,1,0.1),
    'y': np.arange(0,1,0.1),
    'color': [randcol()]*5 + [randcol()]*5,
    'label': ['a']*5 + ['b']*5,
})
overlay = []
for label, df in data.groupby('label'):
    scatter = hv.Scatter(df, kdims=['x', 'y'], vdims=['color', 'label'], label=label).opts(color='color', show_legend=True)
    overlay.append(scatter)
hv.Overlay(overlay)

???

James A. Bednar
@jbednar
hv.Scatter(data, kdims=['x', 'y'], vdims=['color', 'label'], label='label').groupby('color').overlay()
Or, to get better labels, hv.Scatter(data, kdims=['x', 'y'], vdims=['color', 'label'], label='label').groupby('label').overlay()
Thomas Diederen
@Patrickens_gitlab
Thank you! Thats the brevity I was looking for
James A. Bednar
@jbednar
A PR to add that sort of example to the Scatter reference gallery notebook would be appreciated. :-)
You surely aren't the only one to wonder!
Thomas Diederen
@Patrickens_gitlab
@jbednar Ive never done such a thing, but this might be a good place to start
James A. Bednar
@jbednar
It's always a good time to start! And this would be useful; otherwise it will definitely fall off our radar.
David Saroff
@davidsaroff_gitlab
I'm trying to install holoviz in anaconda on a windows 10. When I execute
conda install -c pyviz holoviz
I get the error
ERROR conda.core.link:_execute(700): An error occurred while installing package 'defaults::rise-5.6.1-py37_1'.