These are chat archives for orbitjs/orbit.js

5th
Mar 2016
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:30
@dgeb I decided I was jumping ahead a bit there, we don’t need to split out the "transformation pipeline" / "confirmation process” until we do the optimistic workflow.
    jsonApiSource = new JsonApiSource({ schema: schema });
    store = new Store({ schema: schema });

    store.on('transform', t => queue.add(t));
    queue.on('transform', t => jsonApiSource.transform(t).catch(e => store.deny(t, e)));
    jsonApiSource.on('transform' => store.confirm(t));
I think this is the simplest approach for now. Transforms requested by the UX are confirmed or denied, streamed transforms are always confirmed.
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:36
Note the queue is taking the place of the connectors, given a queue with the API shown though I don’t see a need for a connector as such.
Dan Gebhardt
@dgeb
Mar 05 2016 13:48
@opsb ok, I'm back
so, that last gist is pretty much what I've had in mind
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:49
ok, yeah, there’s certainly the closest to where we are now
Dan Gebhardt
@dgeb
Mar 05 2016 13:49
yes
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:49
ok, well if you’re happy with that I’ll go with that
I’ve been wondering about having factories to set up these common scenarios
but maybe that’s not worth worrying about yet
Dan Gebhardt
@dgeb
Mar 05 2016 13:51
ok, so each store is in charge of maintaining the queue for the source, right? and the queue is internal to the store?
let's leave off factories until we feel good about everything
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:51
sure
so there would be a "queue” in the Store, used to track the outcome of a transform (although it wouldn’t really be a queue)
and then the queue in the gist above would be external
so that gist would currently need to go in an app’s initializer, in the same way that we had to wire up connectors before
Dan Gebhardt
@dgeb
Mar 05 2016 13:54
so doesn't the store need to know about the queue?
in order to pause / clear it
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:55
that can be handled further upstream, because we’re using store.confirm/store.deny the upstream process is independent
all the store is doing is emitting the transform and then waiting to hear if it was successful or not
and when a transform is confirmed it applies it to it’s Cache
Dan Gebhardt
@dgeb
Mar 05 2016 13:58
gotcha
seems good - please proceed :)
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:59
ok, great :)
I’ll get on with that this afternoon
Dan Gebhardt
@dgeb
Mar 05 2016 13:59
thanks much :)
Oliver Searle-Barnes
@opsb
Mar 05 2016 13:59
I was getting a bit stuck on names before, but that’s because I was splitting the confirmation process in 2
this will do us for now though
cool
Dan Gebhardt
@dgeb
Mar 05 2016 14:01
I am just cleaning up the cache now
Oliver Searle-Barnes
@opsb
Mar 05 2016 14:02
great, would be good to start using the transform-builders in the integration tests, put everything together
Dan Gebhardt
@dgeb
Mar 05 2016 14:02
definitely
I'm wondering about support for sparse vs. non-sparse caches
whether we should just fill in sparse details
like relationships
Oliver Searle-Barnes
@opsb
Mar 05 2016 14:04
hmm
Dan Gebhardt
@dgeb
Mar 05 2016 14:04
I think that our use cases for caches have narrowed
Oliver Searle-Barnes
@opsb
Mar 05 2016 14:04
true
this is for transactions right?
that’s the only one I can think of now
Dan Gebhardt
@dgeb
Mar 05 2016 14:05
right - transactions + stores are the two use cases now
I will have to think through some scenarios
Oliver Searle-Barnes
@opsb
Mar 05 2016 14:06
can’t we always allow lazy-loading?
yeah, need to think through how transactions would be used
are we using them for the “editing context” scenario?
Dan Gebhardt
@dgeb
Mar 05 2016 14:08
that is the main use case for transactions I think
anyway, I'll work up some integration tests to think it through
Oliver Searle-Barnes
@opsb
Mar 05 2016 14:09
yeah, ok
Oliver Searle-Barnes
@opsb
Mar 05 2016 16:46
@dgeb so the transform side of things is coming along nicely. I’m updating the tests for Store and see that we need to do something for query. What do you think about the following?
store.on('query', expression => jsonApiSource.query(expression));
Oliver Searle-Barnes
@opsb
Mar 05 2016 17:01
so the Store’s implementation for query would just be
  query(expression) {
    return this.resolve('query').then(() => {
      return this.cache.query(expression);
    });
  },
So this is where our discussions about the QueryResult come into play. To ensure the Cache has been populated with the results of the query before store.query() returns we need to make them available as a result from this.resolve(‘query')
something like:
 query(expression) {
    return this.resolve('query').then((queryResult) => {
      this.cache.transform(queryResult.asTransform());
      return this.cache.query(expression);
    });
  },
Oliver Searle-Barnes
@opsb
Mar 05 2016 17:44
The other option we discussed, which might work well, is to include a pointer to the first transform which includes the results of the query e.g.
  query(expression) {
    return this
      .resolve('query')
      .then(queryResult => this._transformTracker.add(queryResult.lastTransform))
      .then(() => this.cache.query(expression));
  },
this way we can reuse the same blocking mechanism as store.transform()
Dan Gebhardt
@dgeb
Mar 05 2016 18:07
let's say that Source#query just returns a promise (which optionally resolves to a QueryResult)
and that before the promise is resolved, all the transforms to the source that result from the query are emitted via transformed
Dan Gebhardt
@dgeb
Mar 05 2016 18:17
I know that we've discussed returning the transforms in the QueryResult, but I think it might be a simplifiction to emit any data received by a source via on('transform')
^ @opsb
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:18
We can probably get away with that for now. It won’t be guaranteed that the results are in the Cache but if source.on(’transform') is wired directly into the Store then I think we’re ok.
Dan Gebhardt
@dgeb
Mar 05 2016 18:19
should the coordinator be delaying the query promise until the results are in the cache?
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:19
It would be the Store as I’m currently implementing it, but that’s the intent of my last gist
(there’s no coordinator now)
Dan Gebhardt
@dgeb
Mar 05 2016 18:20
sorry, that's what I meant :P
I still have the concepts separate in my head
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:20
ha
I think consolidating these things in the Store does make sense
Dan Gebhardt
@dgeb
Mar 05 2016 18:21
yes, it's a good start
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:21
just pushed my wip up to orbitjs/orbit.js#265
Dan Gebhardt
@dgeb
Mar 05 2016 18:22
great :)
so, we don't really need to return queryResult.lastTransform
since the Store is issuing the query
and receiving emitted transforms
it knows the last transform emitted, right?
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:23
how would it know the last transform?
I mean, it knows the last transform it received, but it doesn’t know when it’s received the transforms for a particular query
A contrived example, imagine there’s a component that delays the emitted transforms from the JSONAPI Source reaching the Store for 10s
Dan Gebhardt
@dgeb
Mar 05 2016 18:25
yeah, fair enough
so we should always return a queryResult that references a Transform
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:26
yeah, I think that’s all we need it to contain
I like the idea of having a queryResult though as it allows us scope for accessing data/transforms etc.
Dan Gebhardt
@dgeb
Mar 05 2016 18:26
yes
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:28
ok, so this means we need transformable._transform() to return that information
sorry, source._query(exp)
Dan Gebhardt
@dgeb
Mar 05 2016 18:29
right
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:29
I do like that this unifies how we block for transforms and queries
Dan Gebhardt
@dgeb
Mar 05 2016 18:29
yes, I agree
so a query's results could always be in a single transform
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:30
yes
but perhaps we just return all of them?
might save us some changes later on
Dan Gebhardt
@dgeb
Mar 05 2016 18:31
QueryResults#transform or QueryResults#transformId
references or ids?
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:31
hmm
I’m leaning towards transform
Dan Gebhardt
@dgeb
Mar 05 2016 18:31
seems good
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:31
seeing as we send it to transformable.transform(transform)
Dan Gebhardt
@dgeb
Mar 05 2016 18:32
otherwise we'd have to retrieve it from a transform map in the Store
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:32
true
Dan Gebhardt
@dgeb
Mar 05 2016 18:33
alright, by ref is good
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:33
so what about returning transform vs [transforms] for _query() ?
Dan Gebhardt
@dgeb
Mar 05 2016 18:33
we're still returning QueryResult, which contains a transform or transforms, right?
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:34
yeah, sorry, so I’m assuming queryable.query() returns QueryResult
I’m just wondering about queryable._query()
Dan Gebhardt
@dgeb
Mar 05 2016 18:34
I see
I think QueryResult could in theory contain raw data as well (at some future point)
so we would need to return QueryResult from _query to support that
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:36
unless we converted the transforms into the data in queryable.query()
we could have different hooks, but I suspect we might be overthinking this now ;)
Dan Gebhardt
@dgeb
Mar 05 2016 18:37
ha! we have that tendency
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:37
ha, I’m hopeful that once we have it released we might be dealing with some more concrete scenarios :)
Dan Gebhardt
@dgeb
Mar 05 2016 18:37
yes
too much up in the air
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:38
ok, so we return QueryResult, we could perhaps have a couple of factory methods
QueryResult.fromTransforms() QueryResult.fromData()
Dan Gebhardt
@dgeb
Mar 05 2016 18:38
seems fine
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:39
yeah, ok, I’ll implement QueryResult.fromTransforms() for now
ok, I think that’s it then
Dan Gebhardt
@dgeb
Mar 05 2016 18:39
ok, nice
btw, I have chosen to simplify the Cache for now
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:40
interesting, how?
Dan Gebhardt
@dgeb
Mar 05 2016 18:40
eliminating the sparse and fallback options - caches are now simply sparse
and the fallback implementation was a leaky abstraction
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:41
yeah, was a bit fuzzy on that area
so how do you populate a sparse cache?
Dan Gebhardt
@dgeb
Mar 05 2016 18:41
paths are filled in as needed
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:42
so it’s lazy loaded?
Dan Gebhardt
@dgeb
Mar 05 2016 18:42
so if you add a model with a type that doesn't yet exist, the type segment gets filled in
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:42
ah, interesting
Dan Gebhardt
@dgeb
Mar 05 2016 18:42
and if a relationship is added, it doesn't require a full representation of the record in place
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:43
one of the things on my list is populating records when links are added before the records
ha!
nice
Dan Gebhardt
@dgeb
Mar 05 2016 18:43
I have become very unhappy with the defaults used for normalization
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:44
that was something I had to workaround before (with the firebase source, inverse links would often need to be populated before the record arrived)
Dan Gebhardt
@dgeb
Mar 05 2016 18:44
gotcha
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:44

I have become very unhappy with the defaults used for normalization

which defaults are we talking about?

Dan Gebhardt
@dgeb
Mar 05 2016 18:45
I don't like the idea of adding a bunch of default empty values for relationships
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:45
right
Dan Gebhardt
@dgeb
Mar 05 2016 18:46
it is a misleading representation of the data
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:46
we did discuss using meta data to indicate a relationship’s state
Dan Gebhardt
@dgeb
Mar 05 2016 18:46
which by nature of the client-side, will always be incomplete
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:46
true
Dan Gebhardt
@dgeb
Mar 05 2016 18:47

we did discuss using meta data to indicate a relationship’s state

yes, we can still do this

because it may be in a partially loaded state
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:47
yeah, that is a slightly thorny one
Dan Gebhardt
@dgeb
Mar 05 2016 18:48
this is just like filling in a single record at the top-level by type, like planet, and assuming that's the complete set
we need metadata for every "set"
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:49
right
Dan Gebhardt
@dgeb
Mar 05 2016 18:50
alright I better work on shipping :shipit:
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:50
yep, ok, I’ll get that QueryResult change in and continue with the json api integration testing
Dan Gebhardt
@dgeb
Mar 05 2016 18:51
ok, nice :)
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:52
BTW, I’ve removed the existing integration tests, they’re all fairly tightly coupled to the old architecture
Can you think of any that might be worth converting? Or are you happy to just add the ones we’ve discussed and go from there?
Dan Gebhardt
@dgeb
Mar 05 2016 18:52
we can always reach back into the history if needed
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:53
yep
Dan Gebhardt
@dgeb
Mar 05 2016 18:53
I would prefer to clear out code than comment it in general
Oliver Searle-Barnes
@opsb
Mar 05 2016 18:53
for sure
Oliver Searle-Barnes
@opsb
Mar 05 2016 20:01
@dgeb just wanted to confirm: this approach with QueryResult means that Queryable will emit the transforms included in the QueryResult before queryable.query() returns, is this what you had in mind?
(currently it just returns the data)
ah, my mistake, I see it does in fact settle the transforms
I’ll change that to emit?
(seeing as blocking is in the Store)
Dan Gebhardt
@dgeb
Mar 05 2016 20:04
I think it does need to emit the transforms (via transformed)
and return the same transforms in the QueryResult
so then the Store can confirm that them
before returning results
Oliver Searle-Barnes
@opsb
Mar 05 2016 20:06
ok, great, that’s me clear
thanks
Dan Gebhardt
@dgeb
Mar 05 2016 20:06
thanks for checking :)
Oliver Searle-Barnes
@opsb
Mar 05 2016 20:55
@dgeb another point I wanted to confirm: cache.query() returns a set in the form {recordId: record}. WRT QueryResult though queryResult.data should be in the normalized schema format?
Dan Gebhardt
@dgeb
Mar 05 2016 21:02
@opsb by default the results would be normalized (say, finding all records), but it's possible that it could be sorted, reformatted, reduced, etc.
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:02
yeah, ok
so that leaves a bit of a conundrum, Cache needs to return results in both formats, for use in the Store and for use in the MemorySource
if the result can be anything though the MemorySource can’t just convert the result format, it doesn’t know what it will be
Dan Gebhardt
@dgeb
Mar 05 2016 21:04
I don't follow
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:06
perhaps I’m confused, am I correct in thinking
memorySource.query() => normalized format
store.query() => set format
?
or would store.query() return the normalized format as well?
(assuming a record query)
Dan Gebhardt
@dgeb
Mar 05 2016 21:09
I believe that the same data should be returned in either case
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:09
the UX needs a starting point for traversal though
e.g.
emberSource.query(qb => qb.recordsOfType(‘pluto’)).then(records => {})
Dan Gebhardt
@dgeb
Mar 05 2016 21:10
if you request all records by type, without a sort order, then we return a set
if you request all records by type, sorted, then we return an array
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:11
what about sideloaded data?
I suppose this is where my confusion lies
I assumed source.query() => {planet: {}, moon: {}}
Dan Gebhardt
@dgeb
Mar 05 2016 21:12
that would be retrieved via a separate query in general
it would be propagated via transforms
so you only make one remote query
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:13

that means that

queryResult.transforms
queryResult.data

would contain different information

Dan Gebhardt
@dgeb
Mar 05 2016 21:13
yes
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:13
the transforms would include everything, the data only the “root"
Dan Gebhardt
@dgeb
Mar 05 2016 21:14
I believe that is ok
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:14
yeah, I think that’s fine in principle, I wonder if it’s the naming that has confused me though, data implies everything to me (sideloaded information as well)
Dan Gebhardt
@dgeb
Mar 05 2016 21:15
hmmm
even in the json:api spec, the data member returns the primary data
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:15
ah, interesting
Dan Gebhardt
@dgeb
Mar 05 2016 21:15
and included returns sideloaded
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:15
right
Dan Gebhardt
@dgeb
Mar 05 2016 21:16
if you request that data sorted, then we should return it as an array
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:16
ok, I suppose my mental model was that store.query() and source.query() served different purposes
store.query() to find the primary resources, source.query() to collect the data including sideloaded data
but I can’t see a problem with the approach you’re describing
it is probably simpler
Dan Gebhardt
@dgeb
Mar 05 2016 21:17
consistency ftw
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:17
yep
ok, done
Dan Gebhardt
@dgeb
Mar 05 2016 21:17
:)
I am sorry about the cache taking so long - the test suite is huge and I've hit a failure
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:18
so what does jsonApiSource.query currently return?
ha, no worries, I’ve made a fair few failures for myself now ;)
Dan Gebhardt
@dgeb
Mar 05 2016 21:19
ha!
ah, so the JSONAPISource _query method needs a more advanced query parser
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:20
right, was just looking, seems to not have any operators :)
Dan Gebhardt
@dgeb
Mar 05 2016 21:21
it might be easiest for you to work using a "remote" memory source for now
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:21
right
ok, I’ll leave JSONAPISource alone then
Dan Gebhardt
@dgeb
Mar 05 2016 21:21
thanks
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:22
I think I’ll need to comment out a few tests due to the introduction of the QueryResult
Dan Gebhardt
@dgeb
Mar 05 2016 21:22
unfortunately, the queryEvaluator is too simplistic - it needs a couple passes - one to parse and another to eval
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:22
right, I skirted that one with the rethinkdb source
it only interpets a single record result or an array of records
Dan Gebhardt
@dgeb
Mar 05 2016 21:23
gotcha ... I was wondering
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:23
yeah, I just managed to avoid implementing a generalised solution
I was thinking about adding the separate interpretation phase onto each operator
Dan Gebhardt
@dgeb
Mar 05 2016 21:24
yeah, I need that here
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:24
right
Dan Gebhardt
@dgeb
Mar 05 2016 21:24
I've got my work cut out for me :/
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:25
yeah, right
Dan Gebhardt
@dgeb
Mar 05 2016 21:25
going to walk the dog ... bbs
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:25
cool
hmm, if a QueryResult can contain anything, how do we convert to transforms?
Oliver Searle-Barnes
@opsb
Mar 05 2016 21:32
using the normalized format does make that simple (just a bunch of addRecord operations)
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:08
We seem to have differing requirements
source.query() => records/links to be added to Store
store.query() => primary resources, sorting, reductions etc.
If QueryResult contains a superset of the two scenarios it’s going to put extra burden on both the Store and various Source implementations.
It’s also worth bearing in mind that we don’t actually need a MemorySource for the base scenario with the JSONAPI source.
(I’m thinking a little about workload here, we can always add MemorySource, satisfying source.query() later)
Dan Gebhardt
@dgeb
Mar 05 2016 22:12

if a QueryResult can contain anything, how do we convert to transforms?

each source needs to convert results to transforms as the query is processed

sources also need to emit those same transforms
via transformed
as the query is processed
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:13
I think that’s going to require a fair bit of work on cache.query()
(not to mention cache.liveQuery())
Dan Gebhardt
@dgeb
Mar 05 2016 22:15
so the primary use case for querying sources is to return / emit transforms
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:15
yes
Dan Gebhardt
@dgeb
Mar 05 2016 22:15
it seems that it is fine if sources aren't even able to return raw data too
maybe that's an option passed into the source - to say that you want data too
and some sources don't support that
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:16

it seems that it is fine if sources aren't even able to return raw data too

agreed

maybe that's an option passed into the source - to say that you want data too
where would that information be consumed? the store currently ignores it
Dan Gebhardt
@dgeb
Mar 05 2016 22:17
right - the only time it would be used is to query a source directly
say you want to get some raw json
it is not the standard use case at all
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:18
that seems out of scope really
Dan Gebhardt
@dgeb
Mar 05 2016 22:18
we can ignore for now
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:18
ok
Dan Gebhardt
@dgeb
Mar 05 2016 22:19
I don't think we need the distraction :P
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:19
so, the main reason I bring this up though is the Cache, emitting transforms would require reworking it
ha!
Dan Gebhardt
@dgeb
Mar 05 2016 22:19
ok, that's true
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:20
I don’t really see that we need to
it’s purpose is to serve the UX
while the MemorySource is nice to have we don’t need it yet
Dan Gebhardt
@dgeb
Mar 05 2016 22:20
essentially we want QueryResults.transforms from most sources and QueryResult.data from the cache / memory
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:21
yes
Dan Gebhardt
@dgeb
Mar 05 2016 22:21
so maybe we have an issue with reusing the method query for both
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:22
yeah, they do seem to be different use cases
Dan Gebhardt
@dgeb
Mar 05 2016 22:22
agreed
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:22
retrieve is perhaps more accurate for sources
or load even
remembering that we’re dealing with a JSON abstraction, something like retrieve might make more sense
dgeb @dgeb thinking ....
Dan Gebhardt
@dgeb
Mar 05 2016 22:29
for tonight, we can stick with query on both and think on it
Source#querycan return promisified transforms
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:30
yes, I’m more concerned about the information requirements
ok
Dan Gebhardt
@dgeb
Mar 05 2016 22:30
and Cache#query returns data
(no transforms)
does that make sense?
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:31

Source#querycan return promisified transforms

would we integrate the transforms directly, in the store.query implementation, or wait until they’ve been delivered via “connectors etc."

?
(been mulling this one a little today)
I’ve been considering the possibility that information is always pulled into the cache first and then distributed to interested sources from there (perhaps after confirmation)
this would make sense for local storage for instance, it would simply listen to confirmed transforms coming out of the cache
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:37
I don’t think that would limit us
We could still do things like dynamic source selection for queries matching criteria for instance
we’re splitting the process into information “collection" and “distribution"
(collection => querying, distribution => localstorage etc. )
Dan Gebhardt
@dgeb
Mar 05 2016 22:40
I agree with that distinction, but I'm unclear on your proposal
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:42
so currently all sources have a permanent connection to the store
but we could always return the transforms to the store after calling one of
store.transform
store.query
store.liveQuery
if we were to do that we don’t need the permanent conncetions
As I understand it the reason for this approach not being taken originally was that we needed to allow the transforms to propagate to other sources first.
Now that we have a log based system all we need to ensure is that the transforms arrive in the correct sequence
(to each Source)
This means we no longer need to settle before returning the information to the Store
The other Sources can listen for confirmed transforms on the Store and integrate later
Dan Gebhardt
@dgeb
Mar 05 2016 22:50
essentially we have a hub/spoke relationship between the store/sources
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:50
yes
although potentially we could still have sources chained together in the spokes
(as it were)
Dan Gebhardt
@dgeb
Mar 05 2016 22:51
I don't know if we should still have promise-aware events with this approach tbh
i.e. events can be emitted but maybe never block
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:53
I think that would give us extra freedom
error handling needs to be considered though
Dan Gebhardt
@dgeb
Mar 05 2016 22:55
I think that if the sources don't maintain their own queues, then promise-aware events fall apart
perhaps
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:56
my impression is that the promise-aware events were there to allow the sources to communicate processing state to the front end
Dan Gebhardt
@dgeb
Mar 05 2016 22:57
yes, they enable chaining sources for blocking transforms
all the way to/from the front end if necessary
Oliver Searle-Barnes
@opsb
Mar 05 2016 22:57
that turned out to be problematic though, because as soon as you have more than one process initiating events (e.g. UX and a remote stream) the blocking promise chains collide with each other
Dan Gebhardt
@dgeb
Mar 05 2016 22:58
right
I suppose that one solution could be two-way connectors that include confirmation
i.e. what you're working on in the store
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:01
yes, although confirmation is really as simple as
remoteSource.on(’transform’, t => store.confirm(t))
now that we have a log I think one way event streams should work well
but error handling is very up in the air (in either case really)
Dan Gebhardt
@dgeb
Mar 05 2016 23:05
so let's explore our possible approaches here
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:05
ok
Dan Gebhardt
@dgeb
Mar 05 2016 23:06
A) a single store acting as coordinator for X number of sources, in which the store maintains queues + confirmation logs for each source
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:09
B) a single store acting as coordinator for X number of sources, where each source maintains it’s own queue using the observable spec (this would allow the queue to be emptied on error so external control wouldn’t be necessary)
(note we don’t necessarily need RxJS to use the observable spec - we could implement something simple ourselves)
Dan Gebhardt
@dgeb
Mar 05 2016 23:10
C) decentralized connectors that maintain queues + confirmation logs between sources
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:11
(C) is the approach I’ve been looking at today
Dan Gebhardt
@dgeb
Mar 05 2016 23:11
that's an interesting idea to use observables without Rx in (B)
what do you think of (C) so far?
it seems the closest to "original" orbit
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:12
so far I think it’s good
I’m definitely happy to run with it if there’s your preferred option at the moment
we do need to figure out the error handling hooks though :)
Dan Gebhardt
@dgeb
Mar 05 2016 23:15
absolutely (re: error handling)
can we keep a MemorySource inside the Store to generalize the problem?
so that connectors are exclusively between sources
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:17
we have the query/something problem at the moment (with the MemorySource)
Dan Gebhardt
@dgeb
Mar 05 2016 23:18
right
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:19
(basically what you saw earlier)
in that scenario the “connectors” are really just events being sent from one source to the store
there is a queue though, which sits between the store and the json api source
Dan Gebhardt
@dgeb
Mar 05 2016 23:24
so in this case where are the confirmation logs?
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:25
the store has a TransformTracker, it isn’t a log, but it’s responsible for blocking until a transform is confirmed
and confirm / deny above
Dan Gebhardt
@dgeb
Mar 05 2016 23:27
are promises returned to the transform events?
e.g. t => queue.add(t)
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:30
Hmm, computer's stopped responding... (on the phone)
They're just transforms
Dan Gebhardt
@dgeb
Mar 05 2016 23:30
so queue.add(t) doesn't return a promise?
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:31
No, the transform tracker creates a deferred promise which blocks until transformTracker.confirm(transform) is called
I'll give you some links when my computer comes back to life...
Dan Gebhardt
@dgeb
Mar 05 2016 23:33
ok, no worries ... I get what you're saying
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:33
ok, back now
Dan Gebhardt
@dgeb
Mar 05 2016 23:34
:)
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:34
these macs don’t last like they used to...
Dan Gebhardt
@dgeb
Mar 05 2016 23:34
ha!
so what are the remaining issues to consider with this approach? the two usages of query + error handling?
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:35
yes
the basic approach seems solid
it still makes use of promises at the sources, but we’re free to use just events as well (they seem interchangeable)
so I think the query issue is ok, as long as we handle store.query and source.query separately (with a new name TBD)
so really it’s just the error handling
Dan Gebhardt
@dgeb
Mar 05 2016 23:37
maybe source.processQuery ?
not so great I know :/
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:38
ha, yeah, I’m struggling as well
Dan Gebhardt
@dgeb
Mar 05 2016 23:39
well, just use something different for now
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:39
yeah, easy to change
Dan Gebhardt
@dgeb
Mar 05 2016 23:39
yeah we'll revisit very soon
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:39
are you happy to remove MemorySource temporarily?
there’s not much to it anyway ;)
definitely nice to have, but it is going to be a chunk of work
Dan Gebhardt
@dgeb
Mar 05 2016 23:40
isn't it used by LocalStorageSource?
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:40
I’m just thinking about your timeline really
ah, true
these seem like things we can come back to though (if we’re being ruthless on scope for this next release)
this is a query issue actually
maybe we keep MemorySource / LocalStorageSource but just don’t include query yet
Dan Gebhardt
@dgeb
Mar 05 2016 23:42
seems fine
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:42
that would satisfy the LocalStorageSource requirements anyway (we just read the whole thing at startup)
Dan Gebhardt
@dgeb
Mar 05 2016 23:42
right
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:45
ok, you got enough in the tank for discussing error handling or shall we come back to it?
Dan Gebhardt
@dgeb
Mar 05 2016 23:45
I've got to break for a bit, and then sort out my cache problem
maybe we can talk it over fresh tomorrow
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:45
yeah, ok
I should have some time tomorrow
Dan Gebhardt
@dgeb
Mar 05 2016 23:46
alright - thanks for slogging this out with me
no pressure though :)
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:46
I’m pretty keen to pull this all together :)
it’s been a while
Dan Gebhardt
@dgeb
Mar 05 2016 23:46
same!
alright ttys
Oliver Searle-Barnes
@opsb
Mar 05 2016 23:47
I think if we can get json api out the door then the rethinkdb stuff should follow fairly easily
yep, ttys
Dan Gebhardt
@dgeb
Mar 05 2016 23:47
:)