Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    and λy.P alpha-equivalent to λz.P'
    otherwise a naive substitution like λy.yx[x := xy] would have the free y in x := xy bound/captured by the lambda
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    that's the relevant high-level concept here
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    I will probably have to read some examples somewhere to understand the concept and "algorithm"
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    well, that's just a good reference for how it should work
    how it does work is a whole other story
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    def create_fgraph():
        a = at.scalar('a')
        b = at.exp(a); b.name = 'b'
        c = at.log(b); c.name = 'c'
        d = c + 5; d.name = 'd'
        e = at.log(d); e.name = 'e'
        f = e - 3; f.name = 'f'
        nodes = dict(a=a, b=b, c=c, d=d, e=e, f=f)
        fg = FunctionGraph(inputs=[a], outputs=[f], clone=False)
        return nodes, fg
    
    # Can always replace one variable by an earlier one
    nodes, fg = create_fgraph()
    fg.replace_validate(nodes['c'], nodes['b'])
    
    nodes, fg = create_fgraph()
    fg.replace_validate(nodes['e'], nodes['c'])
    
    # Can never replace one variable by a later one
    nodes, fg = create_fgraph()
    # This would enter an inifinite loop!
    # fg.replace_validate(nodes['b'], nodes['c'])
    
    # Unless it is the output variable, but then it gives
    # an invalid cyclical graph
    nodes, fg = create_fgraph()
    fg.replace_validate(nodes['e'], nodes['f'])
    
    aesara.dprint(fg)
    # Elemwise{sub,no_inplace} [id A] 'f'   0
    #  |Elemwise{sub,no_inplace} [id A] 'f'   0
    #  |TensorConstant{3} [id B]
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]

    :point_up: Edit: ```python
    def create_fgraph():
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    e = at.log(d); e.name = 'e'
    f = e - 3; f.name = 'f'
    nodes = dict(a=a, b=b, c=c, d=d, e=e, f=f)
    fg = FunctionGraph(inputs=[a], outputs=[f], clone=False)
    return nodes, fg

    Can always replace one variable by an earlier one

    nodes, fg = create_fgraph()
    fg.replace(nodes['c'], nodes['b'])

    nodes, fg = create_fgraph()
    fg.replace(nodes['e'], nodes['c'])

    Can never replace one variable by a later one

    nodes, fg = create_fgraph()

    fg.replace(nodes['b'], nodes['c'])

    Unless it is the output variable, but then it gives

    an invalid cyclical graph

    nodes, fg = create_fgraph()
    fg.replace_validate(nodes['e'], nodes['f'])

    aesara.dprint(fg)

    Elemwise{sub,no_inplace} [id A] 'f' 0

    |Elemwise{sub,no_inplace} [id A] 'f' 0

    |TensorConstant{3} [id B]

    ```

    :point_up: Edit: ```python
    def create_fgraph():
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    e = at.log(d); e.name = 'e'
    f = e - 3; f.name = 'f'
    nodes = dict(a=a, b=b, c=c, d=d, e=e, f=f)
    fg = FunctionGraph(inputs=[a], outputs=[f], clone=False)
    return nodes, fg

    Can always replace one variable by an earlier one

    nodes, fg = create_fgraph()
    fg.replace(nodes['c'], nodes['b'])

    nodes, fg = create_fgraph()
    fg.replace(nodes['e'], nodes['c'])

    Can never replace one variable by a later one

    Produces an invalid cyclical graph

    nodes, fg = create_fgraph()
    fg.replace(nodes['b'], nodes['c'])

    Unless it is the output variable, but then it gives

    an invalid cyclical graph

    nodes, fg = create_fgraph()
    fg.replace(nodes['e'], nodes['f'])

    aesara.dprint(fg.outputs)

    Elemwise{sub,no_inplace} [id A] 'f' 0

    |Elemwise{sub,no_inplace} [id A] 'f' 0

    |TensorConstant{3} [id B]

    ```

    Ricardo Vieira
    @ricardov94:matrix.org
    [m]

    Last code dump I promise

    #%%
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    
    fg = FunctionGraph(inputs=[a], outputs=[c], clone=False)
    fg.replace(b, d)
    aesara.dprint(fg.outputs)
    # Elemwise{log,no_inplace} [id A] 'c'
    #  |Elemwise{add,no_inplace} [id B] 'd'
    #    |Elemwise{log,no_inplace} [id A] 'c'  <-- CYCLICAL
    #    |TensorConstant{5} [id C]
    
    #%%
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    
    fg = FunctionGraph(inputs=[a], outputs=[b], clone=False)
    fg.replace(b, d)
    aesara.dprint(fg.outputs)
    # Elemwise{add,no_inplace} [id A] 'd'
    #  |Elemwise{log,no_inplace} [id B] 'c'
    #  | |Elemwise{exp,no_inplace} [id C] 'b'
    #  |   |a [id D]
    #  |TensorConstant{5} [id E]

    Does it make sense that the replacement of b -> d works when b is the output of the FunctionGraph (second half) but not when it is not (first half)?

    Is this just a corner case I am hitting, and they should both fail/ or succeed?
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]

    :point_up: Edit: Last code dump I promise

    #%%
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    
    fg = FunctionGraph(inputs=[a], outputs=[c], clone=False)
    fg.replace(b, d)
    aesara.dprint(fg.outputs)
    # Elemwise{log,no_inplace} [id A] 'c'
    #  |Elemwise{add,no_inplace} [id B] 'd'
    #    |Elemwise{log,no_inplace} [id A] 'c'  <-- CYCLICAL
    #    |TensorConstant{5} [id C]
    
    #%%
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    
    fg = FunctionGraph(inputs=[a], outputs=[b], clone=False)
    fg.replace(b, d)
    aesara.dprint(fg.outputs)
    # Elemwise{add,no_inplace} [id A] 'd'
    #  |Elemwise{log,no_inplace} [id B] 'c'
    #  | |Elemwise{exp,no_inplace} [id C] 'b'
    #  |   |a [id D]
    #  |TensorConstant{5} [id E]

    Does it make sense that the replacement of b -> d "works" when b is the output of the FunctionGraph (second half) but not when it is not (first half)?
    By works I mean that it produces an acyclical graph

    :point_up: Edit: Is this just a corner case I am hitting, and should both fail/ or succeed?
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    one minute
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    when you replaced the output b in the latter case, it basically cleared the entire FunctionGraph and replaced it with another graph/output that also happened to reference b
    so no issue there
    remember, these steps need to happen in a fixed sequence of events
    and the actual replacement steps are different depending on whether or not the replaced term is an output
    in that case, the thing being changed is FunctionGraph.outputs
    when a replacement is made on something that is not just a FunctionGraph output, like in the former case, an Apply node is updated in-place
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    specifically, I believe it's Apply.inputs that's updated in-place
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    anyway, there's no real cycle detection at this level, so it's completely up to the caller to not introduce them
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    Okay, now things are clicking a bit more for me. The (aesara) problem is not with the recursive expression, but somehow its identity?
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    fg = FunctionGraph(inputs=[a], outputs=[d], clone=False)
    
    # Cannot do this
    # fg.replace(b, c)
    
    # But can do this
    new_c = c.owner.op(*c.owner.inputs); new_c.name = 'new_c'
    fg.replace(b, new_c)
    
    aesara.dprint(fg.outputs)
    Is there a reason why fg.replace(b, c) should behave differently than fg.replace(b, new_c)?
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    :point_up: Edit: Things are still not clicking entirely. The (aesara) problem I was facing before is not with the recursive expression, but somehow its identity?
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    fg = FunctionGraph(inputs=[a], outputs=[d], clone=False)
    
    # Cannot do this
    # fg.replace(b, c)
    
    # But can do this
    new_c = c.owner.op(*c.owner.inputs); new_c.name = 'new_c'
    fg.replace(b, new_c)
    
    aesara.dprint(fg.outputs)
    :point_up: Edit: Things are still not clicking entirely. The (aesara) problem I was facing before is not with the recursive expression, but somehow its identity?
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    fg = FunctionGraph(inputs=[a], outputs=[d], clone=False)
    
    # Cannot do this
    # fg.replace(b, c)
    
    # But can do this
    new_c = c.owner.op(*c.owner.inputs); new_c.name = 'new_c'
    fg.replace(b, new_c)
    
    aesara.dprint(fg.outputs)
    # Elemwise{add,no_inplace} [id A] 'd'   
    #  |Elemwise{log,no_inplace} [id B] 'c'   
    #  | |Elemwise{log,no_inplace} [id C] 'new_c'   
    #  |   |Elemwise{exp,no_inplace} [id D] 'b'   
    #  |     |a [id E]
    #  |TensorConstant{5} [id F]
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    :point_up: Edit: Things are still not clicking entirely. The (aesara) problem I was facing before is not with the recursive expression, but somehow its identity?
    a = at.scalar('a')
    b = at.exp(a); b.name = 'b'
    c = at.log(b); c.name = 'c'
    d = c + 5; d.name = 'd'
    fg = FunctionGraph(inputs=[a], outputs=[d], clone=False)
    
    # Cannot do this
    # fg.replace(b, c)
    
    # But can do this
    # new_c = at.log(b); new_c.name = 'new_c'
    new_c = c.owner.op(*c.owner.inputs); new_c.name = 'new_c'
    fg.replace(b, new_c)
    
    aesara.dprint(fg.outputs)
    # Elemwise{add,no_inplace} [id A] 'd'   
    #  |Elemwise{log,no_inplace} [id B] 'c'   
    #  | |Elemwise{log,no_inplace} [id C] 'new_c'   
    #  |   |Elemwise{exp,no_inplace} [id D] 'b'   
    #  |     |a [id E]
    #  |TensorConstant{5} [id F]
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    I seem to be digging around the same ground as in this old issue: Theano/Theano#5482
    1 reply
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    Yeah, I know that, I am trying to understand what are the commonalities. Between fgraph.replace, function.givens, and clone_replace, each one seems to follow it's own unique logic
    clone_replace seems completely broken for more than one replacement
    a = at.scalar('a')
    b = a + 1; b.name = 'b'
    c = b + 1; c.name = 'c'
    d = c + 2; d.name = 'd'
    e = d + 2; e.name = 'e'
    f = e + 1; f.name = 'f'
    
    out = aesara.clone_replace(f, replace={b: at.cos(a), d: at.sin(c)})
    aesara.dprint(out)
    # Elemwise{add,no_inplace} [id A] 'f'   
    #  |Elemwise{add,no_inplace} [id B] 'e'   
    #  | |Elemwise{sin,no_inplace} [id C] ''   
    #  | | |Elemwise{add,no_inplace} [id D] 'c'   
    #  | |   |Elemwise{add,no_inplace} [id E] 'b'   
    #  | |   | |a [id F]
    #  | |   | |TensorConstant{1} [id G]
    #  | |   |TensorConstant{1} [id H]
    #  | |TensorConstant{2} [id I]
    #  |TensorConstant{1} [id J]
    I understand d: at.sin(c) erases the first update, but then clone_replace with multiple updates only works in very specific cases where the replacements are not nested at all
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    one of the biggest differences is that FunctionGraph.replace is an in-place update of Apply.inputs that operates on lambdas-like objects and has some lambda-relevant considerations
    it also keeps track of term relationships within the lambda
    and uses those
    the other approaches don't
    they simply walk the graph in topological order and make replacements sequentially
    and the substitutions have little to no "awareness" of each other
    e.g. one substitution can re-introduce a substituted variable
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    that's a result of the substitution order
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    Yeah, I am finding that. It would be really nice to document this a bit more in detail. I was really missing a doc page with a non trivial example of how to manipulate graphs and how the different ways differ and gotchas. This was my semi-systematic way of looking at these: https://colab.research.google.com/drive/1jmrkyYiYP_Z0IsERCpN_tdX-GddpNgZU?usp=sharing
    clone_replace also does something weird, where it first replaces the keys of the replacements to dummy variable in a graph and then replaces those dummy variables by the desired "new values". This seems to be a work around to allow for updates that use variables that depend on the variable being replaced
    But this seems to break multiple replacements as fair as I can understand, as in my last example above with sin and cos replacements
    Ricardo Vieira
    @ricardov94:matrix.org
    [m]
    Also, some of these seem still very relevant so it might be useful to discuss them: Theano/Theano#5483
    brandonwillard
    @brandonwillard:matrix.org
    [m]
    most of those aren't actually bugs
    just confusions caused by a lack of documentation and understanding
    with some attemps to help via more warnings/messages
    i.e. not really solutions to the underlying problems
    the closest one to a solution is the request for documentation