Please use Stack Overflow with the #dask tag for usage questions and github issues for bug reports
solve
operation across the last 2 dimensions, i.e. given A
and B
I want to do C[i,:,:]=solve(A[i,:,:], B[i,:,:])
for all i in the leading dimension... I tried the below but it seems to be really slow (slower than numpy) - does anyone know what I'm doing wrong/what I could do better? Sorry if this was the wrong place to ask.C = da.apply_gufunc(np.linalg.solve, "(i,j),(i,k)->(j,k)", A, B, vectorize=True,output_dtypes=A.dtype)
scale
and then read it back then scale up/down from that?
@mrocklin Do you think a maximum limit of workers which can be used by a submitted job would be useful, Although I see that we already have a param for providing a set of workers.
On a broader idea worker pools, something like this could be rather useful.
Can these two behaviours be achieved currently?
map_blocks(func_that_returns_a_debug_string)
)
map_blocks
assumes that I'm mapping an array on to another array, which doesn't match these use cases well
I use pytest to test my application. Since it relies on dask, I need to have a local cluster as a fixture.
I tried to do it myself and close the cluster on the fixture cleanup. Even though it works, I get a warning with more than 100 lines because dask tries to communicate with the cluster after it has been closed.
I use pytest to test my application. Since it relies on dask, I need to have a local cluster as a fixture.
I tried to do it myself and close the cluster on the fixture cleanup. Even though it works, I get a warning with more than 100 lines because dask tries to communicate with the cluster after it has been closed.
I've also tried to use the client fixture from distributed.utils_test, but that gives me the following error
file /Users/ruiloureiro/.virtualenvs/tarsier/lib/python3.7/site-packages/distributed/utils_test.py, line 561
@pytest.fixture
def client(loop, cluster_fixture):
E fixture 'loop' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, client, cov, doctest_namespace, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.`
Hey all, my application requires me to launch tasks from within other tasks, like the following
def a():
# ... some computation ..
def b():
# ... some computation ..
def c():
client = get_client()
a = client.submit(a)
b = client.submit(b)
[a,b] = client.gather([a,b])
return a+b
client = get_client()
res = client.submit(c)
However, I would like to have access to the intermediate results a
and b
, but only c shows up in client.futures
.
Is there a way to tell dask to keep the results for a
and b
?
Thank you
task_states[task] = executor.submit(
self.run_task,
task=task,
state=task_state,
... snipped ...
)