These are chat archives for orbitjs/orbit.js

12th
Feb 2016
Oliver Searle-Barnes
@opsb
Feb 12 2016 06:56

@dgeb I started to consider the question, how would the UI bind to the results of a multi resource query. Take for example:

const planets = store.query(
  qb.recordsOfType('planet')
    .belongingToStar(theSun)
    .moons(m => m.withWater()) // criteria on related records
);

{{#each planets as |planet|}}
  <h1>{{planet.name}}</h1>
  <ul>
  {{#each planet.moons as |moon|}}
    <li>{{moon.name}}</li>
  {{/each}}
  </ul>
{{/each}}

Currently we don't have any way of doing this, planet.moons will always include all records from the Cache. It strikes me that what we need is similar to how transactions currently work, a temporary Store that will contain the filtered information for all of the resources.

const filteredStore = store.createTransaction(
  qb.recordsOfType('planet')
    .belongingToStar(theSun)
    .moons(m => m.withWater())
)

const moonsWithWater = filteredStore.query(qb.recordsOfType('moon'));

This solves the question, what representation should we use to represent the results of a query? The answer is that it's a Source (the MemorySource inside the Transaction). Transforms are all that’s necessary to transport the information, the important thing is that they’re not just integrated with a single Cache.

There are a few questions around this, regarding the identity map and flows for mutations but now that ember-orbit's data flow is one way I don't see any blockers.

Dan Gebhardt
@dgeb
Feb 12 2016 20:50
@opsb sorry, been tied up all day, just reading now ...

what representation should we use to represent the results of a query? The answer is that it's a Source

this seems consistent with what I was saying yesterday about returning a cache of normalized data with a query

although, as you say, that normalized data could be conveyed just with transforms
Oliver Searle-Barnes
@opsb
Feb 12 2016 20:58
yeah, I was looking to see how it would work end to end, and realised that we probably have enough structures to express a solution already.
Dan Gebhardt
@dgeb
Feb 12 2016 20:59
that's true - I have no doubt that we can express a solution
I'm just concerned about the interfaces
Oliver Searle-Barnes
@opsb
Feb 12 2016 20:59
sure
Dan Gebhardt
@dgeb
Feb 12 2016 20:59
I feel like we have a design problem here
Oliver Searle-Barnes
@opsb
Feb 12 2016 20:59
yeah, I agree
Dan Gebhardt
@dgeb
Feb 12 2016 20:59
having query return an array of transforms is no bueno
the source could emit those transforms via an event or stream
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:00
yeah, but how about
const querySource = source.query(query)
where querySource contains the results
Dan Gebhardt
@dgeb
Feb 12 2016 21:01
well, we could construct a QueryResults class
that internalizes a cache
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:01
How would it differ from a Source?
Dan Gebhardt
@dgeb
Feb 12 2016 21:01
I can't imagine calling it querySource tbh
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:02
It could just be a MemorySource
A Source is just a container of information after all
It seems to match the requirements for query results
I should clarify one point though
Dan Gebhardt
@dgeb
Feb 12 2016 21:02
ok, I see where you're going
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:02
I’m considering 2 stages of filtering
Dan Gebhardt
@dgeb
Feb 12 2016 21:03
say more
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:03
1) Query for a graph/Source of results
2) Bind an ember-orbit array to a type in those results
That way the UI can traverse the results
The canonical store then just becomes a special case, which integrates resuts from all queries
Dan Gebhardt
@dgeb
Feb 12 2016 21:06
hmmm ... so you would traverse results via a store and models, right?
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:06
yeah
Dan Gebhardt
@dgeb
Feb 12 2016 21:06
and have the option to merge those results with the main store
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:06
right
Dan Gebhardt
@dgeb
Feb 12 2016 21:18
I think it's worth poking around the mismatch between say, a query that requests a sorted array, and a result that is simply a blob of unsorted data
I agree that the blob of unsorted data is important to return
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:19
hmm, that’s a good point
perhaps that’s handled at the second stage of filtering
Dan Gebhardt
@dgeb
Feb 12 2016 21:19
you are thinking about this holistically, which is good
but each method / return val needs to make sense
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:20
sure, I suppose I’m considering it in terms of information required for a query, data + filtering/sorting
Dan Gebhardt
@dgeb
Feb 12 2016 21:21
yeah, I was getting at this yesterday as well
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:22
there could be a single representation, but consider that Sources can have any structure
So we could use them to wrap a query result structure
Expressing both forms of data in a single structure is effectively what GraphQL does.
Dan Gebhardt
@dgeb
Feb 12 2016 21:24
well, I suppose the QueryResult could re-issue the same query on its cache if people want results from it in say, a sorted array form
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:25
yes, that’s what I would consider the second stage
Dan Gebhardt
@dgeb
Feb 12 2016 21:25
const result = restSource.query();
result.data(); // returns array / set / etc
allow for that, but don't incur the overhead automatically
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:27
The data in that case would contain sets for for multiple types right?
i.e. it would include sideloaded data for instance
Dan Gebhardt
@dgeb
Feb 12 2016 21:28
the data inside the cache would need to include everything returned from the server
but the data() query would only include results from the same query on the result's cache
so that's a bit nuanced
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:29
I was confused by:
result.data(); // returns array / set / etc
Dan Gebhardt
@dgeb
Feb 12 2016 21:29
oh sorry
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:29
right
Dan Gebhardt
@dgeb
Feb 12 2016 21:29
I just meant that it would return the data sorted, presented like the query asked for
like if the same query were issued on a cache (which it would be)
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:30
right
ok, so it’s the lossless aspect that you’re trying to solve
i.e. potentially we can just dish the results from the server straight up to the UI with no further querying in memory
Dan Gebhardt
@dgeb
Feb 12 2016 21:32
so this is closer than I've felt to a good solution
but we're definitely not there entirely
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:32
yeah, feels like a lossless solution could be achieved with something close, but I’m not sure
I think the lossless solution would have to be a graph of results
well, tree I should say
I can’t see how else you could express the information without losing something
I suppose the difficulty we’re having is that orbit has always normalized at the edges
i.e. the sources always convert information into the normalized form before interacting with anything else in orbit
Dan Gebhardt
@dgeb
Feb 12 2016 21:36
well, oql is infinitely flexible - it's just a question of what we want to support in orbit-common queries
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:36
and that isn’t sufficient for expressing query results
true
Dan Gebhardt
@dgeb
Feb 12 2016 21:36
I am not convinced that we want to support something like graphql in orbit-common
but someone could write oql processors for it and all the sources
and it would work
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:37
I only reference that because it’s a structure that expresses the same information
the results of a query
Dan Gebhardt
@dgeb
Feb 12 2016 21:37
true
it's one example
of a format
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:37
with sorting etc.
Dan Gebhardt
@dgeb
Feb 12 2016 21:37
right
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:37
sure, I’m not proposing we use it
Dan Gebhardt
@dgeb
Feb 12 2016 21:38
it's a good format - I just think it's unnecessary right now
I would like to be confident that we could support it
without changing how we process queries
that seems a good litmus test to me
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:40
yeah, that does seem like a good goal
Dan Gebhardt
@dgeb
Feb 12 2016 21:41
and I think we need to treat the query evaluator more like a compiler
with a parse step if needed
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:41
Because OQL is infinitely extensible we need to capture the results in a container that we can stream to/from
Dan Gebhardt
@dgeb
Feb 12 2016 21:42
yeah
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:42

and I think we need to treat the query evaluator more like a compiler

that’s why I mentioned ‘interpreter'

Dan Gebhardt
@dgeb
Feb 12 2016 21:42
right, I agree
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:43
So perhaps we use Sources / Transforms for the interfaces, we just need to figure out a new query result structure for orbit-common?
Dan Gebhardt
@dgeb
Feb 12 2016 21:47
yeah, are you ok with QueryResult?
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:48
I think we should be looking at a different json structure
I still think that a Source is the correct container
Dan Gebhardt
@dgeb
Feb 12 2016 21:48
so what if QueryResult extends Source ?
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:49
yeah, that would seem to make sense
Dan Gebhardt
@dgeb
Feb 12 2016 21:49
so that it contains the original query and can return it evaluated against its internal cache
soooooooo
this has interesting implications for LiveQueries, right?
a live query result could also be represented with a QueryResult, right?
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:50
just trying to imagine how that would look, in terms of a UI binding to it
Dan Gebhardt
@dgeb
Feb 12 2016 21:56
well, first things first, I'll have a go at getting QueryResultworking with both the Cache and JSONAPISource
Oliver Searle-Barnes
@opsb
Feb 12 2016 21:56
internets a little patchy here...

a live query result could also be represented with a QueryResult, right?

yes, absolutely!

So it seems to come down to when we normalise/denormalize
The UI is denormalized
It could bind directly to a QueryResult
Rather than having to run queries against a normalized store
Dan Gebhardt
@dgeb
Feb 12 2016 22:00
I think the cache in QueryResult is normalized
it has to be really
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:00
How would you represent ordering?
Dan Gebhardt
@dgeb
Feb 12 2016 22:00
well data() may be denormalized
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:00
I think there should be a Source per representation
Dan Gebhardt
@dgeb
Feb 12 2016 22:00
result.data() could return an ordered array, or a sum, or anything
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:01
i.e. a Source for a normalized form, another for a QueryResult (json structure)
then we can express mappings between them as streams of Transforms
potentially we could wrap these up into higher level interfaces though
Dan Gebhardt
@dgeb
Feb 12 2016 22:03
wouldn't the structure of the cache in a QueryResult need to match that of a normalized MemorySource's cache so that queries could be issued against it?
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:03
but then we have a lossy solution
a lossless solution would have to be denormalized
Having thought about it though I’m not sure we need to solve that problem now
It seems like orbit has enough primitives to create a lossless solution
But I’m not sure that’s orbit-common, perhaps it’s something equivalent
Dan Gebhardt
@dgeb
Feb 12 2016 22:08
I am unclear on the value of allowing denormalized data to flow through any of orbit's sources
I can see wanting to query a remote source for denormalized data
but orbit doesn't need to make sense of that result
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:08
That is the GraphQL use case
Dan Gebhardt
@dgeb
Feb 12 2016 22:09
yes
I suppose
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:09
I’m happy parking that one though for now though :)
I think we could cover it without any dramatic changes to orbit anyway
We both seem to be headed in the same direction with the QueryResult idea anyway, that was what I was trying to solve with my earlier comments.
Dan Gebhardt
@dgeb
Feb 12 2016 22:11
yeah
the situations where graphql will work with QueryResult are limited to when data can be normalized before it's denormalized via a query on the result's cache
so we need to figure out if orbit cares at all about returning random blobs of denormalized data directly
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:13
right
I think it needs to to allow traversal
oh sorry, missed the last part of that sentence
no, I think we can focus on keeping denormalized data at the edges
with normalised data inside orbit
Dan Gebhardt
@dgeb
Feb 12 2016 22:14
yeah, seems good to me too
let's park that one, as you said
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:14
agreed
Dan Gebhardt
@dgeb
Feb 12 2016 22:14
we have enough on our plate :)
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:14
yep
Dan Gebhardt
@dgeb
Feb 12 2016 22:15
alright, gotta run for now, but I'll work on QueryResult soon
Oliver Searle-Barnes
@opsb
Feb 12 2016 22:16
ok, need to think about how the UI is going to bind to it I think, the bit where I’m fuzzy is how to express queries for the two stages of filtering
(as I’m calling them)
anyway, that seems like good progress
Dan Gebhardt
@dgeb
Feb 12 2016 22:16
:+1: