by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 21:59
    joegoldbeck edited #4446
  • Jan 31 2019 21:56

    martindurant on master

    Fix relative path parsing on wi… (compare)

  • Jan 31 2019 21:56
    martindurant closed #4445
  • Jan 31 2019 21:55
    martindurant commented #4445
  • Jan 31 2019 21:54
    Dimplexion commented #4445
  • Jan 31 2019 21:47
    joegoldbeck opened #4446
  • Jan 31 2019 21:41
    TomAugspurger commented #4361
  • Jan 31 2019 21:38
    holoneo starred dask/dask
  • Jan 31 2019 21:30
    Mdhvince commented #4361
  • Jan 31 2019 20:52
  • Jan 31 2019 20:49
    mrocklin commented #2497
  • Jan 31 2019 20:22
    mrocklin opened #2497
  • Jan 31 2019 20:18
    mrocklin closed #4444
  • Jan 31 2019 20:18
    mrocklin commented #4444
  • Jan 31 2019 20:17

    mrocklin on 1.25.3

    (compare)

  • Jan 31 2019 20:17

    mrocklin on master

    bump version to 1.25.3 (compare)

  • Jan 31 2019 20:12
    martindurant commented #4445
  • Jan 31 2019 18:52

    mrocklin on master

    bump version to 1.1.1 (compare)

  • Jan 31 2019 18:52

    mrocklin on 1.1.1

    (compare)

  • Jan 31 2019 18:38
    martindurant commented #4445
Jim Crist-Harif
@jcrist
Is it useful? :shrug:. Was interesting to write though.
Was mostly an experiment to see how speedy python serialization could get without schemas.
Martin Durant
@martindurant
@mrocklin : I am looking at https://www.youtube.com/playlist?list=PLRtz5iA93T4PQvWuoMnIyEIz1fXiJ5Pri - where is the dask online tutorial
Matthew Rocklin
@mrocklin
jakirkham
@jakirkham
There use to be an example in the Distributed docs about good/bad ways to use Client.submit and avoid sending large amounts of data through the Client, but I can't find it atm. Does anyone know where this is? Is it still in the docs?
Julia Signell
@jsignell
jakirkham
@jakirkham
Thanks Julia! Yeah that's what I came up with too. I thought there was something like this as well (though I could be misremembering)
# bad
f1 = c.submit(sum, numpy.arange(1_000_000))

# good
f2a = c.submit(numpy.arange, 1_000_000)
f2b = c.submit(sum, f2a)
Matthew Rocklin
@mrocklin
@jcrist a dask-yarn question if you're interested: https://stackoverflow.com/questions/63070444/dask-cant-recognize-workers-on-emr
Julia Signell
@jsignell
I'm up for another friday check in if you are interested @jrbourbeau.
Philipp Rudiger
@philippjfr
We're planning to release Bokeh 2.2 some time next week. Given the breakages we had in previous releases it would be great if someone on the Dask end could do some basic smoke testing once we tag an RC release some time tomorrow.
Matthew Rocklin
@mrocklin
@jsignell or @jacobtomlinson if you have time perhaps?
Thanks for the warning @philippjfr !
Philipp Rudiger
@philippjfr
Bokeh 2.2.0rc1 should now be available from PyPI (using --pre) and bokeh/label/dev on conda.
Julia Signell
@jsignell
Yep I can take a look :)
Julia Signell
@jsignell
I just realized that there can be per-worker dashboards but I can't quite figure out how to see them. From the /info/main/workers.html page I get links like: http://127.0.0.1:8787/proxy/39943/127.0.0.1/status but those just 404.
I instantiated my cluster like
cluster = LocalCluster(dashboard=True)
client = Client(cluster)
client
Martin Durant
@martindurant
In that case I suppose it would be http://127.0.0.1:39943/… , but would be good to know why the proxy didn’t work
Julia Signell
@jsignell
http://127.0.0.1:39943/status also 404s so I don't think it's the proxy. I think I am not getting per-worker dashboards
Martin Durant
@martindurant
Try an extra worker as a CLI process (dask-worker 127.0.0.1:8786); I’m not sure what the local-cluster defaults for the workers are
Julia Signell
@jsignell
Martin Durant
@martindurant
Yep; there are a lot of non-obvious defaults in LocalCluster
Julia Signell
@jsignell
ok thanks! I commented out that line and now I have it running :)
Martin Durant
@martindurant
The worker should also have a /sitemap.json endpoint, listing all available endpoints
(in my PR branch)
-hint- please someone merge the PR, or comment!
Julia Signell
@jsignell
I didn't see anything glaring with the bokeh rc @philippjfr thanks for menting it here!
Julia Signell
@jsignell
Actually I am getting an error on the worker profile page. I need to check that it works in released bokeh though
Julia Signell
@jsignell
yeah not a regression
GUEROUDJI Amal
@GueroudjiAmal
hey everyone,
i am wondering to know if there is any way to access data in some process' memory from dask , from example accessing a in-memory hdf5 file that resides in an mpi process memory ?
Martin Durant
@martindurant
Has anyone ever seen a test run fail after the test itself passed? https://github.com/dask/dask/pull/6528/checks?check_run_id=1003538655#step:11:33
Benjamin Zaitlen
@quasiben
@martindurant no. But looking around it looks like an NPM issue. Maybe something messed in up messaging back to GH ?
Martin Durant
@martindurant
GH action fix is in goanpeca/setup-miniconda#64
Matthew Rocklin
@mrocklin
I made it half-way through my github e-mail backlog! The best part was finding that one issue included the solution for another issue :)
Julia Signell
@jsignell
I am not sure how to help bring dask/dask#6520 to a conclusion. Would appreciate someone chiming in.
Tom Augspurger
@TomAugspurger
Monthly meeting starts now!
Martin Durant
@martindurant
@mrocklin @jacobtomlinson : what did you use for after-tutorial feedback?
Matthew Rocklin
@mrocklin
I think that @jacobtomlinson had a google form?
Unrelatedly, I just learned that github has a discussion feature: https://github.com/PrefectHQ/prefect/discussions
Julia Signell
@jsignell
I just learned that yesterday as well :) I think it's in beta. But could be a good solution to the discourse dilema
Matthew Rocklin
@mrocklin
+1
Thomas J. Fan
@thomasjpfan
It looks like github is very responsive to joining the discussions beta by commenting here: https://github.community/t/can-one-apply-for-github-discussions-beta/121120
Matthew Rocklin
@mrocklin
Ooh, thanks for the pointer @thomasjpfan ! I've made a request for access there.
Martin Durant
@martindurant
Issue sprint starting soon? I expect we’ll be on whereby. I’ll be fresh with an Earl Grey tea. I will be taking a short break midway to present at Anaconda demo-friday.
Julia Signell
@jsignell
:+1:
Jim Crist-Harif
@jcrist
Be there momentarily
Matthew Rocklin
@mrocklin
Aw, I miss demo friday
Martin Durant
@martindurant
We should invite you sometime!
Matthew Rocklin
@mrocklin
I'm in!