These are chat archives for ipython/ipython

30th
Aug 2016
Bas Nijholt
@basnijholt
Aug 30 2016 14:10

I'm encountering a mysterious ipyparallel bug I think (@minrk).
I have this function:

def plot_phase(dview, lview, p, W, H, dim, a=10):
    syst_str = 'syst = NS_infinite_2D_3D(a={}, W={}, H={}, dim={})'.format(a, W, H, dim)
    dview.execute(syst_str, block=True)

    dview['p'] = p

    Bs = np.linspace(0, 1, 50)
    mus = np.linspace(0.1, 15, 50)

    Es = lview.map_async(lambda x: find_gap(syst, p.update(B_x=x[0], mu_sm=x[1]), 201), 
                         product(Bs, mus))
    Es.wait_interactive()
    return Es.result()

When I define this function in a py-file and import it and call it, I get this error:

[135:apply]: 
---------------------------------------------------------------------------NameError                                 Traceback (most recent call last)<string> in <module>()
/home/basnijholt/anaconda3/lib/python3.5/site-packages/ipyparallel/client/remotefunction.py in <lambda>(f, *sequences)
    248             if _mapping:
    249                 if sys.version_info[0] >= 3:
--> 250                     f = lambda f, *sequences: list(map(f, *sequences))
    251                 else:
    252                     f = map
/home/basnijholt/Dropbox/Work/short_jj_majorana/Code/shortjunction.py in <lambda>(x)
    526 
    527     if async_parallel:
--> 528         Es = lview.map_async(lambda x: find_gap(syst, p.update(B_x=x[0], mu_sm=x[1]), num_k), 
    529                              product(Bs, mus))
    530         Es.wait_interactive()
NameError: name 'syst' is not defined

However, when I define the function in the notebook, everything works fine!
What is happening?!

Min RK
@minrk
Aug 30 2016 14:12
You are running into this
The issue is that since lambda x: is defined in a module, when it looks up syst it looks in that module, not in __main__. If you defined that function in the notebook, it will look it up in __main__, which is the difference.
You can use the interactive() decorator from ipyparallel to tell IPython to treat the function as if it were defined interactively, which means it will look for undefined names in __main__ instead of the module.
Bas Nijholt
@basnijholt
Aug 30 2016 14:30
Thanks for the quick answer @minrk! However, when putting the @interactive decorator on top of my function I get the following error:
if async_parallel:
    610         decay_lengths = lview.map_async(
--> 611             lambda x: slowest_evan_mode(syst, p.update(B_x=x[0], mu_sm=x[1], mu_sc=x[1]), a), product(Bs, mus))
    612 
    613         decay_lengths.wait_interactive()

NameError: name 'product' is not defined
Is the solution to just import everything inside the function?
Min RK
@minrk
Aug 30 2016 14:30
you will have to do one or the other, since the function only gets one namespace to look in
You can also use parallel.Reference('syst') as an argument passd to the function, which will always resolve in the interactive environment
e.g. view.apply(lambda x: x, Reference('syst'))
References are always resolved in the main module
Bas Nijholt
@basnijholt
Aug 30 2016 14:36
That sounds like a better solution.
So just
syst = ipyparallel.Reference('syst')
Es = lview.map_async(lambda x: find_gap(syst, p.update(B_x=x[0], mu_sm=x[1]), num_k), 
                             product(Bs, mus))
Min RK
@minrk
Aug 30 2016 14:38
Not quite, I think it's a bit trickier than that with map ratter than apply.
Bas Nijholt
@basnijholt
Aug 30 2016 14:38
It also doesn't work, hehe.
Min RK
@minrk
Aug 30 2016 14:47
Here's an example that works:
import ipyparallel as ipp

def testf(dview, lbview, A):
    dview.execute('remote_y = 2', block=True)
    # resolve y in the remote namespace by adding a second argument to map
    # with a Reference on the name
    ys = [ipp.Reference('remote_y')] * len(A)
    return lbview.map(lambda x,y: x * y, A, ys)
The Reference has to be an argument to the function.
So we resolve the namespace by giving the mapped function an extra argument that is the Reference to the remote value
that way x is the main item, y is the remote value, and any other names are resolved in the module
Bas Nijholt
@basnijholt
Aug 30 2016 14:52
And why wouldn't this not work:
def testf(dview, lbview, A):
    dview.execute('remote_y = 2', block=True)
    return lbview.map(lambda x: x * ipp.Reference('remote_y'), A)
Bas Nijholt
@basnijholt
Aug 30 2016 15:13
The example that you gave works for my code.
This code is going to be published with a paper, so I prefer to keep it as clean as possible. Would you know a more elegant solution?
Min RK
@minrk
Aug 30 2016 16:05
Because ipp.Reference is evaluated at the wrong time in that example
you could use sys.modules['__main__'].remote_y, which would do the same thing.