Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Yuvi Panda
    @yuvipanda
    cluster object doesn't seem to have either methods
    conda-forge's dask is at v0.15? or am I misreading https://anaconda.org/conda-forge/dask?
    (sorry for all the dumb questions!)
    yuvipanda @yuvipanda tries unpinning dask in environment file
    Martin Durant
    @martindurant
    Says 2.8.1
    Yuvi Panda
    @yuvipanda
    aaah
    it says that for linux noarch
    but has 4 badges saying 0.15
    image.png
    my bad
    Yuvi Panda
    @yuvipanda
    hmm, LocalCluster has a .logs and shows me logs
    while KubeCluster does not
    And LocalCluster does worki
    Yuvi Panda
    @yuvipanda
    hmm, I guess this is how people new to JupyterHub feel :D
    I'm gonna go figure out how to get logs to go somewhere not stderr
    Matthew Rocklin
    @mrocklin
    maybe get_logs?
    There is, I believe some logs method there
    Yuvi Panda
    @yuvipanda
    @mrocklin yeah! except there isn't much there, so i was trying to set it to debug. AFter doing so in ~/.dask/config.yaml, nothing there either
    Yuvi Panda
    @yuvipanda
    hmm, I reverted back to an Image used in a different hub that I know is working
    so this is confusing
    Yuvi Panda
    @yuvipanda
    aaargh, it was a typo! My RoleBinding referred to a ClusterRole and not a Role
    why wasn't this in the k8s audit logs?!
    anyway, thanks for coming along to this ride
    Ray Bell
    @raybellwaves
    Hi. First time running dask on Azure using k8s and helm.
    My notes are here: https://sites.google.com/view/raybellwaves/courses/azure-vms#h.p_6Mb0lF20RMqI
    I believe i'm ~90% there just having trouble getting the dashboard
    image.png
    Do I need to do anything to get the dashboard setup? i.e. copy the URL (which URL?) to the search bar in the Dask extension?
    Ray Bell
    @raybellwaves
    For reference
    image.png
    Ok. Found using the scheduler external IP
    image.png
    Ray Bell
    @raybellwaves
    Just hoping to pin in next to my notebook using the Dask extension
    slavarazbash
    @slavarazbash
    Hi! The Enterprise Data Science Architecture Conference focuses on how to properly productionise data science solutions at scale. Dask is a tool that I have personally used to get the job done. Most Dask users would be interested in seeing how large companies productionise machine learning solutions. 27th March 2020 is a great time to visit Melbourne, Australia for a unique and high quality conference. I invite you view our speakers list at https://edsaconf.io and reserve your place because we have a unique mix of speakers.
    dayu
    @dayuoba
    hi gus, i'm new to dask, i want to konw if there is some tutorials about deploying dask jobs on k8s in native way? i tried with official docs, but i can not even run the demo successfully
    i've containerized a demo python job , and deploy a pod on my k8s cluster, it always retiring the other workers.
    i've also created rbac for the pod
    simaster123
    @simaster123
    Hello - I'm struggling to resolve the issue I posted here: dask/dask#5634. Any chance that there's anyone here open to a short consulting gig to help me debug it?
    Ray Bell
    @raybellwaves
    @raybellwaves Updating link to my notes: https://sites.google.com/view/raybellwaves/cheat-sheets/azure-vms#h.p_6Mb0lF20RMqI
    AwesomeCap
    @AwesomeCap
    Hi, how does dask work with Amdahl's Law?
    iu.png
    AwesomeCap
    @AwesomeCap
    What is the parallel portion?
    JoranDox
    @JoranDox
    @AwesomeCap I think that depends on your code
    if you write code that doesn't need to shuffle/sync across dask nodes and is "embarrasingly parallellisable", you'll go into the realm of 95% maybe
    if you write code that sequentially goes over your data row per row you'll be at 0%
    that said, the scheduler can be a bottleneck for really big task graphs
    but I'm not sure if that's always the case, we haven't scaled to the size where it made sense to look into that
    AwesomeCap
    @AwesomeCap
    do you think dask will scale for a more effective scheduler, maybe sometime in the future? or is it more "nice-to-have"? :)
    Martin Durant
    @martindurant
    The performance of the scheduler is always being optimised… There have been specific attempts to reimplement in cython or other, but be assured that the often quoated “1ms overhead per task” is pessimistic.
    codecnotsupported
    @codecnotsupported
    I tried to make a SSHCluster with a tunnel but it would seem Dask doesn't play nicely with Asyncssh. https://bpaste.net/show/K7XOG :"got Future <Future pending> attached to a different loop".