Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Yuvi Panda
    @yuvipanda
    aha, if I run cluster.scale(1), after a while I get:
    Task exception was never retrieved
    future: <Task finished coro=<_wrap_awaitable() done, defined at /opt/conda/envs/geostack/lib/python3.7/asyncio/tasks.py:596> exception=AssertionError()>
    Traceback (most recent call last):
      File "/opt/conda/envs/geostack/lib/python3.7/asyncio/tasks.py", line 603, in _wrap_awaitable
        return (yield from awaitable.__await__())
      File "/opt/conda/envs/geostack/lib/python3.7/site-packages/distributed/deploy/spec.py", line 42, in _
        assert self.status == "running"
    AssertionError
    tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <zmq.eventloop.ioloop.ZMQIOLoop object at 0x7f0da548f550>>, <Task finished coro=<SpecCluster._correct_state_internal() done, defined at /opt/conda/envs/geostack/lib/python3.7/site-packages/distributed/deploy/spec.py:284> exception=AssertionError()>)
    Traceback (most recent call last):
      File "/opt/conda/envs/geostack/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
        ret = callback()
      File "/opt/conda/envs/geostack/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
        future.result()
      File "/opt/conda/envs/geostack/lib/python3.7/site-packages/distributed/deploy/spec.py", line 317, in _correct_state_internal
        await w  # for tornado gen.coroutine support
      File "/opt/conda/envs/geostack/lib/python3.7/site-packages/distributed/deploy/spec.py", line 42, in _
        assert self.status == "running"
    AssertionError
    is there a way to make logs very verbose?
    I can tell that no calls are being made to kubernetes
    so it's somewhere in dask
    Martin Durant
    @martindurant
    :)
    Yuvi Panda
    @yuvipanda
    nothing seems to help unfortunately. I think what's happening is the scheduler process isn't starting, but I actually don't know if the scheduler is an external process or not
    Matthew Rocklin
    @mrocklin
    @yuvipanda you may want cluster.pods() and cluster.logs()
    (if you're using dask-kubernetes)
    Yuvi Panda
    @yuvipanda
    I am
    cluster object doesn't seem to have either methods
    conda-forge's dask is at v0.15? or am I misreading https://anaconda.org/conda-forge/dask?
    (sorry for all the dumb questions!)
    yuvipanda @yuvipanda tries unpinning dask in environment file
    Martin Durant
    @martindurant
    Says 2.8.1
    Yuvi Panda
    @yuvipanda
    aaah
    it says that for linux noarch
    but has 4 badges saying 0.15
    image.png
    my bad
    Yuvi Panda
    @yuvipanda
    hmm, LocalCluster has a .logs and shows me logs
    while KubeCluster does not
    And LocalCluster does worki
    Yuvi Panda
    @yuvipanda
    hmm, I guess this is how people new to JupyterHub feel :D
    I'm gonna go figure out how to get logs to go somewhere not stderr
    Matthew Rocklin
    @mrocklin
    maybe get_logs?
    There is, I believe some logs method there
    Yuvi Panda
    @yuvipanda
    @mrocklin yeah! except there isn't much there, so i was trying to set it to debug. AFter doing so in ~/.dask/config.yaml, nothing there either
    Yuvi Panda
    @yuvipanda
    hmm, I reverted back to an Image used in a different hub that I know is working
    so this is confusing
    Yuvi Panda
    @yuvipanda
    aaargh, it was a typo! My RoleBinding referred to a ClusterRole and not a Role
    why wasn't this in the k8s audit logs?!
    anyway, thanks for coming along to this ride
    Ray Bell
    @raybellwaves
    Hi. First time running dask on Azure using k8s and helm.
    I believe i'm ~90% there just having trouble getting the dashboard
    image.png
    Do I need to do anything to get the dashboard setup? i.e. copy the URL (which URL?) to the search bar in the Dask extension?
    Ray Bell
    @raybellwaves
    For reference
    image.png
    Ok. Found using the scheduler external IP
    image.png
    Ray Bell
    @raybellwaves
    Just hoping to pin in next to my notebook using the Dask extension
    slavarazbash
    @slavarazbash
    Hi! The Enterprise Data Science Architecture Conference focuses on how to properly productionise data science solutions at scale. Dask is a tool that I have personally used to get the job done. Most Dask users would be interested in seeing how large companies productionise machine learning solutions. 27th March 2020 is a great time to visit Melbourne, Australia for a unique and high quality conference. I invite you view our speakers list at https://edsaconf.io and reserve your place because we have a unique mix of speakers.
    dayu
    @dayuoba
    hi gus, i'm new to dask, i want to konw if there is some tutorials about deploying dask jobs on k8s in native way? i tried with official docs, but i can not even run the demo successfully
    i've containerized a demo python job , and deploy a pod on my k8s cluster, it always retiring the other workers.
    i've also created rbac for the pod
    simaster123
    @simaster123
    Hello - I'm struggling to resolve the issue I posted here: dask/dask#5634. Any chance that there's anyone here open to a short consulting gig to help me debug it?
    Ray Bell
    @raybellwaves