## Where communities thrive

• Join over 1.5M+ people
• Join over 100K+ communities
• Free without limits
##### Activity
Chase Wilson
@Kixiron
I've been working on a little side project that may be interesting for ddlog, it requires updating timely & differential to the latest master and adding some regions to the dataflows to be used to its full effect though
Leonid Ryzhyk
@ryzhyk
Nice! Is this graph for a DDlog-generated dataflow (I don't think anybody writes DD programs like this by hand)?
Chase Wilson
@Kixiron
Yes, this is the rslint graph
Leonid Ryzhyk
@ryzhyk
I notice there are some orphan join nodes.
Also, if we were using named versions of operators recently added to DD, would those names show up in the graph?
Chase Wilson
@Kixiron
Yah, those are kinda funky
Yep! That was part of my goal, named regions and named operators all show up!
If you hover over nodes there's some tooltips and the colors are also customizable (for that map brighter is slower)
There's also some redundant concats in addition to those orphan joins
Leonid Ryzhyk
@ryzhyk
Are orphan joins actually orphan (not sure what that'd even mean) or is this a problem in the visualization tool?
Chase Wilson
@Kixiron
I'm also planning to add ingestion rates (how many records were passed through each channel), but logging changed for that recently so it needs some modifications
I'm assuming it's a bug on my side, I'm still looking into it
Leonid Ryzhyk
@ryzhyk
Likewise, there shouldn't be any dangling arrangements, all arrangements we generate should be used by at least one join.
But yeah, this is incredibly cool, I'd love to use this.

Yep! That was part of my goal, named regions and named operators all show up!

ok, it should be straightforward to switch DDlog to use named operators

Chase Wilson
@Kixiron
I'm hoping it'll help guide me in rewriting our ast -> ddflow compilation system
Chase Wilson
@Kixiron
How do you actually call a relation transformer, I can't find any examples
Leonid Ryzhyk
@ryzhyk
There are some examples. Search for apply in simple.dl, redist.dl.
Chase Wilson
@Kixiron
Oh, so you declare the relations you want the outputs to be "put into" and then have those as outputs in the application?
Leonid Ryzhyk
@ryzhyk
Yes
Transformers are not the most elegant feature. They also don't work inside recursive scopes.
Chase Wilson
@Kixiron
Alright, thanks!
Chase Wilson
@Kixiron
Do you have any idea why Intern<String> wouldn't implement ExchangeData?
error[E0599]: no method named arrange_by_key found for reference &Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32> in the current scope
--> crates\rslint_scope\generated\types\name_in_scope.rs:278:18
|
278 |                   .arrange_by_key()
|                    ^^^^^^^^^^^^^^ method not found in &Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32>
|
::: C:\Users\Chase\.cargo\git\checkouts\differential-dataflow-57c8d81d89be42ac\4431abb\src\collection.rs:41:1
|
41  |   pub struct Collection<G: Scope, D, R: Semigroup = isize> {
|   -------------------------------------------------------- doesn't satisfy _: ArrangeByKey<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, ScopeId, Intern<std::string::String>, i32>
|
::: G:\Users\Chase\Code\Rust\RSLint\crates\rslint_scope\generated\types\internment\internment.rs:82:1
|
82  | / pub struct Intern<A>
83  | | where
84  | |     A: Eq + Send + Sync + Hash + 'static,
85  | | {
86  | |     interned: ArcIntern<A>,
87  | | }
| |_- doesn't satisfy _: differential_dataflow::ExchangeData
|
= note: the method arrange_by_key exists but the following trait bounds were not satisfied:
Intern<std::string::String>: differential_dataflow::ExchangeData
which is required by Collection<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, (ScopeId, Intern<std::string::String>), i32>: ArrangeByKey<timely::dataflow::scopes::Child<'_, S, timely::order::Product<<S as ScopeParent>::Timestamp, u64>>, ScopeId, Intern<std::string::String>, i32>
I tried adding impls of Send + Sync to Intern, but that wasn't it
Oh, Abomonate
Leonid Ryzhyk
@ryzhyk
Just curious: what are you implementing?
Chase Wilson
@Kixiron
I’m rewriting the name resolution part of rslint in ddflow using a transformer
Leonid Ryzhyk
@ryzhyk
Is this something that's not expressible in Datalog? Our main use case for transformers so far have been algorithms that require non-monotonic fixed point computation.
Chase Wilson
@Kixiron
It’s a speed thing, the hand written version is a lot faster and makes significantly less arrangements and will hopefully improve once I finagle dogs^3 into it
Leonid Ryzhyk
@ryzhyk
Have you analyzed where the extra arrangements are coming from? We put some amount of effort into arrangement sharing. It'd be good to know if there are some additional optimizations we could implement.
Chase Wilson
@Kixiron
I'm going to try and tackle it after benchmarking, concurrency and the api refactor all land
Leonid Ryzhyk
@ryzhyk
:thumbsup:
Chase Wilson
@Kixiron
@ryzhyk do you know if the calculus-based delta joins from dogs^3 (not the old propose/validate operators from dogs^3) work correctly in iterative contexts? I’m trying to eliminate some large arrangements with them
Leonid Ryzhyk
@ryzhyk
I haven't tried it, but according to Frank they should
My implementation was done before these existed.
I suspect the performance may not be as good as with propose/validate though.
Chase Wilson
@Kixiron
Alright, thanks!
I’m confused as to what degree that (the current implementation of) worst case joins is incompatible with loops, is it just that Variables can’t contain an alt/neu collection directly?
Leonid Ryzhyk
@ryzhyk
I think it would compile, but according to Frank the algorithm wouldn't work correctly.
Chase Wilson
@Kixiron
Yes, that’s what I was meaning
But using worst case joins simply within an iterative context is fine right?
Leonid Ryzhyk
@ryzhyk
Good question. I don't know.
Chase Wilson
@Kixiron
@ryzhyk How would you feel about me consolidating all of our documentation into an mdBook? It'd allow us to publish to github pages automatically to have hosted docs, give us the ability to have paginated chapters and overall allow the docs to be more easily readable and all that
Leonid Ryzhyk
@ryzhyk
I feel good about any documentation improvements, thanks for offering :) What documentation specifically are you talking about?
Chase Wilson
@Kixiron
Pretty much all of it, combining the tutorial, spec and language-specific tutorials into one book (while extending all of them)
That way users could navigate to what’s relevant to them be it ddlog surface syntax or interop with C
Leonid Ryzhyk
@ryzhyk
Got it. Definitely a good idea. The tutorial's really bloated and poorly structured at this point.
Chase Wilson
@Kixiron
Sounds good, I’ll look into it some
Leonid Ryzhyk
@ryzhyk
Happy to discuss the details. I'm very interested in making the tutorial more accessible.
Chase Wilson
@Kixiron
I don’t have anything as of now but once I get around to it I’ll definitely ping you