These are chat archives for orbitjs/orbit.js

10th
Feb 2016
Dan Gebhardt
@dgeb
Feb 10 2016 15:19
thinking about removing get as a query operator and keeping everything at a higher level (i.e. records, relationships, attributes)
@opsb how would that fit with your rethink source?
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:21
It currently relies on get but using the higher level operators would make more sense. I think it’s only used for attributes anyway.
Dan Gebhardt
@dgeb
Feb 10 2016 15:23
ok, cool - that should be an easy change
I'm also thinking about transforms in the same light
and debating whether to introduce higher level transforms
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:24
how would that work?
and yeah, very easy change for me, in fact it makes a lot more sense because rethinkdb is table oriented (you can’t do get a type for instance)
Dan Gebhardt
@dgeb
Feb 10 2016 15:25
ok great
so I'm considering how queries and transforms could operate on the same higher level, and only translate to the low level patch ops to transform the cache's document
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:27
think I need an example of a higher level transform
Dan Gebhardt
@dgeb
Feb 10 2016 15:27
yeah, fair enough
{op: 'addRecord', value: {type: 'planet', attributes: { name: 'Jupiter' }} }
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:30
ah, right, I thought’s that what you might mean, I have to say my feeling very early on was that higher level might make more sense
in particular with relationships
it means you can describe the denormalized updates with a single operation
It’s a while since I’ve considered it but it does seem to make sense
one complication though
Dan Gebhardt
@dgeb
Feb 10 2016 15:32
right, and I had been fighting it because of the need for general purpose diff transforms - but we have moved to immutable transforms
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:32
when you receive updates from a remote server they’ll be the lower level operations
we could perhaps infer the higher level operation, e.g. this is an add to hasOne, we know the inverse is ...
Dan Gebhardt
@dgeb
Feb 10 2016 15:33
we need loss-less translation from lower to higher level operations
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:34
we’d end up with dupes as we ll
the server would send operations for each side of a relationship for instance
Dan Gebhardt
@dgeb
Feb 10 2016 15:35
are you talking about rethinkdb?
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:35
yeah, or firebase as well
you could filter them based on a rule
i.e. pick a side of the relationship to monitor
tricky though, because different queries might ask for different sides
hmm
Dan Gebhardt
@dgeb
Feb 10 2016 15:37
I wonder if this is a non-issue if both ops are emitted together in an array?
each source has the opportunity to de-dupe as needed
when applying transforms
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:38
maybe we just don’t dedupe?
Dan Gebhardt
@dgeb
Feb 10 2016 15:38
well, the operation processors on the cache will just treat them as no-ops I think
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:38
right
yeah, the cache is where you’d want to optimise
Dan Gebhardt
@dgeb
Feb 10 2016 15:39
so let's worry about this when we have a test case that is problematic
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:40
ok, it matches what the operation processor does anyway I think, it receives an update for one side and updates the other
Dan Gebhardt
@dgeb
Feb 10 2016 15:40
right
so we really need to map out "standard" orbit-common query and transform ops
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:41
yeah
Dan Gebhardt
@dgeb
Feb 10 2016 15:41
and make sure they line up nicely
I will take a stab at this today as I flesh out the memory/jsonapi sources and we can review
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:43
ok, yeah, let’s see where you get to
Dan Gebhardt
@dgeb
Feb 10 2016 15:45
one problem I've been struggling with is representing the results of a query as transforms ("finding" something isn't the same as "adding" it)
and I've started to think that this is not the right problem to solve
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:46
hmm
Dan Gebhardt
@dgeb
Feb 10 2016 15:47
it might be better to allow coordinators to coordinate query and transform results
if both are at the same level
but, on the other hand, sources could emit query results essentially as "replace" operations
i.e. always representing updates
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:49
I suppose I do a see a difference between the two
Dan Gebhardt
@dgeb
Feb 10 2016 15:49
yes
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:49
I see queries as projections from the Sources
Dan Gebhardt
@dgeb
Feb 10 2016 15:50
yes
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:50
with transforms describing changes to synchronise sources
Dan Gebhardt
@dgeb
Feb 10 2016 15:50
sure
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:51
so when you consider them on the same level, where do you see that they aren’t currently?
Dan Gebhardt
@dgeb
Feb 10 2016 15:52
well, transforms deal purely with the normalized form currently
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:52
right
Dan Gebhardt
@dgeb
Feb 10 2016 15:52
and have no concept of "records" yet
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:52
yeah, ok
Dan Gebhardt
@dgeb
Feb 10 2016 15:53
so once transforms and queries have a concept of the higher level abstractions, then a coordinator could make sense of them both
and not force sources to try to convert query results into transforms on a (usually non-existent) internal cache
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:54

and not force sources to try to convert query results into transforms on a (usually non-existent) internal cache

this is the gap for me, when would this be necessary?

Dan Gebhardt
@dgeb
Feb 10 2016 15:54
I think using internal caches on all sources was probably my biggest mistake in the original version of orbit, because it forced the wrong mental model
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:55
right
yeah, I did struggle with that bit
Dan Gebhardt
@dgeb
Feb 10 2016 15:55
so if you query a JSONAPISource, you eventually want the results to end up in your identity map (usually)
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:55
right
Dan Gebhardt
@dgeb
Feb 10 2016 15:56
so if the coordinator only watches transforms, then those results need to be converted into transforms at some point
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:56
ok, right
I see where you’re going now
Dan Gebhardt
@dgeb
Feb 10 2016 15:56
ok
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:57
yeah, I did think about that a little
at the time I thought well, you need to make sure all the Sources are syncrhonised so you shouldn’t bypass the connectors
but yeah, as you say, if the coordinators are aware of this then there’s no problem
it can destribute the transforms as required
Dan Gebhardt
@dgeb
Feb 10 2016 15:58
yeah, exactly
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:58
had to change gears a little bit ;)
Dan Gebhardt
@dgeb
Feb 10 2016 15:58
we have high hopes for our coordinators :)
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:58
ok, yeah, that does sound good
ha!
yeah, we’re kicked the problem down the road
Dan Gebhardt
@dgeb
Feb 10 2016 15:58
to some extent
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:59
but I like the approach
it’s very functional
Dan Gebhardt
@dgeb
Feb 10 2016 15:59
I think it simplifies the sources even more
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:59
we have these consistent data structures (the Sources) and then the coordinators operator over them
Dan Gebhardt
@dgeb
Feb 10 2016 15:59
yes
Oliver Searle-Barnes
@opsb
Feb 10 2016 15:59
that was what I found appealing about orbit in the first place
putting the data structures first
Dan Gebhardt
@dgeb
Feb 10 2016 16:00
right, don't want to lose that focus
so I think this solves the queueing problem as well
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:01
because the coordinator will be taking care of it?
Dan Gebhardt
@dgeb
Feb 10 2016 16:01
since the response from a JSONAPI source is more of a query than an additional transform
the transform is applied, and then additional records may be returned
those additional records should be emitted as a query result
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:02
how are those query results integrated into other Sources?
wouldn’t they need to be transforms?
Dan Gebhardt
@dgeb
Feb 10 2016 16:03
if they truly represent changes to the source, then they should be
but for JSONAPI compound documents, they are just a view into the source
so we need source authors to differentiate
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:04
when you say compound documents are we talking side loading?
Dan Gebhardt
@dgeb
Feb 10 2016 16:04
yeah
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:04
ok
don’t those documents need to be integrated as well?
(the side loaded ones)
Dan Gebhardt
@dgeb
Feb 10 2016 16:05
yes, so this is nuanced, and I'm not sure where to draw the lines
they should essentially be emitted as "found" records
as if they came from a query
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:05
this is why I ended back at a 2 step process, transforms for all data, then query to select it
Dan Gebhardt
@dgeb
Feb 10 2016 16:06
so you would want the JSONAPISource to emit sideloaded records via transforms?
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:07
yeah, my current thinking is that we want to integrate any data that’s received
and then do a query
Dan Gebhardt
@dgeb
Feb 10 2016 16:08
so let's say that you query /articles?include=authors
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:08
but that requires the Cache to rerun the query, even though the query results were identified by the server
Dan Gebhardt
@dgeb
Feb 10 2016 16:08
I see
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:09
having said that, the Cache may have a superset of the server’s results
in which case you’d want to rerun the query
Dan Gebhardt
@dgeb
Feb 10 2016 16:11
so your preference would be for the JSONAPISource to translate any results it receives into transforms (as we're doing now)?
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:13
yeah, I think it’s necessary
so how would it look if they were just emitting via queries?
would the information be integrated into the cache?
or just sent straight to the UI?
Dan Gebhardt
@dgeb
Feb 10 2016 16:14
the coordinator would have to understand the standard queries and apply the results to the cache like it applies transforms it observes
it would basically just be moving that responsibility to the coordinator
I think both approaches are feasible, but this becomes a question of which makes sources cleaner (and I'm still not sure)
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:16
where I’m a bit stuck is that the query results currently don’t include enough information to integrate with the orbit-common structure
Dan Gebhardt
@dgeb
Feb 10 2016 16:17
but if they represent a recordset or recordarray, then they could be integrated
maybe this will become clearer for us both once we start considering both queries + transforms at the higher level
I should start there
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:21
back in a minute
Dan Gebhardt
@dgeb
Feb 10 2016 16:21
sure, gotta head out for a bit myself
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:21
ok
It’s starting to sound like one way data flow, i.e. transforms for writes, queries for updates
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:29
ok, so if I understand this correctly, the coordinators would be responsible for converting query results into transforms for distribution?
Oliver Searle-Barnes
@opsb
Feb 10 2016 16:41
So what would query results look like for compound documents? Currently the query results would only contain information for the root collection. The sideloaded documents are retrieved via additional queries. Potentially query results could be a graph though.
Oliver Searle-Barnes
@opsb
Feb 10 2016 17:12
I suppose the main point is that queries to remote sources include both the record data and the selection specified by the query. Currently we distribute the record data to the Sources via transforms, throw away the selection information and then run the query again against the Cache.
Oliver Searle-Barnes
@opsb
Feb 10 2016 17:18
I think what you’re proposing would require the query results in two different formats, one containing the information from the compound document(root records + sideloaded records) and another which only contained the information for an array in the front end(just the root records).
Oliver Searle-Barnes
@opsb
Feb 10 2016 17:41
So one thing here is that we currently have to block the query at the Source while the transforms settle. Potentially the transforms could be emitted and the query could simply include the last transform id. Then the store could block the query until that transform id had been reached (this could be used with either the current strategy or the one I think you’re proposing).
Dan Gebhardt
@dgeb
Feb 10 2016 19:17
@opsb these are all good questions
I don't think that the query results from a non-memory-source will ever be used directly
i.e. we'll always want results from a Store
so we need to consider how queries for the remote sources will be initiated
we could start by initiating queries directly on remote sources in order to "refresh" any live queries on stores
Dan Gebhardt
@dgeb
Feb 10 2016 19:23
before we go too far, I'd like to do some round trip mapping and stop trying to solve only part of the problem