Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Chase Wilson
    @Kixiron
    dataflow.svg
    I've been working on a little side project that may be interesting for ddlog, it requires updating timely & differential to the latest master and adding some regions to the dataflows to be used to its full effect though
    Leonid Ryzhyk
    @ryzhyk
    Nice! Is this graph for a DDlog-generated dataflow (I don't think anybody writes DD programs like this by hand)?
    Chase Wilson
    @Kixiron
    Yes, this is the rslint graph
    Leonid Ryzhyk
    @ryzhyk
    I notice there are some orphan join nodes.
    Also, if we were using named versions of operators recently added to DD, would those names show up in the graph?
    Chase Wilson
    @Kixiron
    Yah, those are kinda funky
    Yep! That was part of my goal, named regions and named operators all show up!
    If you hover over nodes there's some tooltips and the colors are also customizable (for that map brighter is slower)
    There's also some redundant concats in addition to those orphan joins
    Leonid Ryzhyk
    @ryzhyk
    Are orphan joins actually orphan (not sure what that'd even mean) or is this a problem in the visualization tool?
    Chase Wilson
    @Kixiron
    I'm also planning to add ingestion rates (how many records were passed through each channel), but logging changed for that recently so it needs some modifications
    I'm assuming it's a bug on my side, I'm still looking into it
    Leonid Ryzhyk
    @ryzhyk
    Likewise, there shouldn't be any dangling arrangements, all arrangements we generate should be used by at least one join.
    But yeah, this is incredibly cool, I'd love to use this.

    Yep! That was part of my goal, named regions and named operators all show up!

    ok, it should be straightforward to switch DDlog to use named operators

    Chase Wilson
    @Kixiron
    I'm hoping it'll help guide me in rewriting our ast -> ddflow compilation system
    Chase Wilson
    @Kixiron
    How do you actually call a relation transformer, I can't find any examples
    Leonid Ryzhyk
    @ryzhyk
    There are some examples. Search for apply in simple.dl, redist.dl.
    Chase Wilson
    @Kixiron
    Oh, so you declare the relations you want the outputs to be "put into" and then have those as outputs in the application?
    Leonid Ryzhyk
    @ryzhyk
    Yes
    Transformers are not the most elegant feature. They also don't work inside recursive scopes.
    Chase Wilson
    @Kixiron
    Alright, thanks!
    Chase Wilson
    @Kixiron
    Do you have any idea why Intern<String> wouldn't implement ExchangeData?
    error[E0599]: no method named `arrange_by_key` found for reference `&Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32>` in the current scope
       --> crates\rslint_scope\generated\types\name_in_scope.rs:278:18
        |
    278 |                   .arrange_by_key()
        |                    ^^^^^^^^^^^^^^ method not found in `&Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32>`
        |
       ::: C:\Users\Chase\.cargo\git\checkouts\differential-dataflow-57c8d81d89be42ac\4431abb\src\collection.rs:41:1
        |
    41  |   pub struct Collection<G: Scope, D, R: Semigroup = isize> {
        |   -------------------------------------------------------- doesn't satisfy `_: ArrangeByKey<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, ScopeId, Intern<std::string::String>, i32>`
        |
       ::: G:\Users\Chase\Code\Rust\RSLint\crates\rslint_scope\generated\types\internment\internment.rs:82:1
        |
    82  | / pub struct Intern<A>
    83  | | where
    84  | |     A: Eq + Send + Sync + Hash + 'static,
    85  | | {
    86  | |     interned: ArcIntern<A>,
    87  | | }
        | |_- doesn't satisfy `_: differential_dataflow::ExchangeData`
        |
        = note: the method `arrange_by_key` exists but the following trait bounds were not satisfied:
                `Intern<std::string::String>: differential_dataflow::ExchangeData`
                which is required by `Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32>: ArrangeByKey<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, ScopeId, Intern<std::string::String>, i32>`
    I tried adding impls of Send + Sync to Intern, but that wasn't it
    Oh, Abomonate
    Leonid Ryzhyk
    @ryzhyk
    Just curious: what are you implementing?
    Chase Wilson
    @Kixiron
    I’m rewriting the name resolution part of rslint in ddflow using a transformer
    Leonid Ryzhyk
    @ryzhyk
    Is this something that's not expressible in Datalog? Our main use case for transformers so far have been algorithms that require non-monotonic fixed point computation.
    Chase Wilson
    @Kixiron
    It’s a speed thing, the hand written version is a lot faster and makes significantly less arrangements and will hopefully improve once I finagle dogs^3 into it
    Leonid Ryzhyk
    @ryzhyk
    Have you analyzed where the extra arrangements are coming from? We put some amount of effort into arrangement sharing. It'd be good to know if there are some additional optimizations we could implement.
    Chase Wilson
    @Kixiron
    I'm going to try and tackle it after benchmarking, concurrency and the api refactor all land
    Leonid Ryzhyk
    @ryzhyk
    :thumbsup:
    Chase Wilson
    @Kixiron
    @ryzhyk do you know if the calculus-based delta joins from dogs^3 (not the old propose/validate operators from dogs^3) work correctly in iterative contexts? I’m trying to eliminate some large arrangements with them
    Leonid Ryzhyk
    @ryzhyk
    I haven't tried it, but according to Frank they should
    My implementation was done before these existed.
    I suspect the performance may not be as good as with propose/validate though.
    Chase Wilson
    @Kixiron
    Alright, thanks!
    I’m confused as to what degree that (the current implementation of) worst case joins is incompatible with loops, is it just that Variables can’t contain an alt/neu collection directly?
    Leonid Ryzhyk
    @ryzhyk
    I think it would compile, but according to Frank the algorithm wouldn't work correctly.
    Chase Wilson
    @Kixiron
    Yes, that’s what I was meaning
    But using worst case joins simply within an iterative context is fine right?
    Leonid Ryzhyk
    @ryzhyk
    Good question. I don't know.
    Chase Wilson
    @Kixiron
    @ryzhyk How would you feel about me consolidating all of our documentation into an mdBook? It'd allow us to publish to github pages automatically to have hosted docs, give us the ability to have paginated chapters and overall allow the docs to be more easily readable and all that
    Leonid Ryzhyk
    @ryzhyk
    I feel good about any documentation improvements, thanks for offering :) What documentation specifically are you talking about?
    Chase Wilson
    @Kixiron
    Pretty much all of it, combining the tutorial, spec and language-specific tutorials into one book (while extending all of them)
    That way users could navigate to what’s relevant to them be it ddlog surface syntax or interop with C
    Leonid Ryzhyk
    @ryzhyk
    Got it. Definitely a good idea. The tutorial's really bloated and poorly structured at this point.
    Chase Wilson
    @Kixiron
    Sounds good, I’ll look into it some
    Leonid Ryzhyk
    @ryzhyk
    Happy to discuss the details. I'm very interested in making the tutorial more accessible.
    Chase Wilson
    @Kixiron
    I don’t have anything as of now but once I get around to it I’ll definitely ping you