These are chat archives for orbitjs/orbit.js

10th
Mar 2016
Dan Gebhardt
@dgeb
Mar 10 2016 01:34
ok, thanks much!
Oliver Searle-Barnes
@opsb
Mar 10 2016 07:37
@dgeb I took a quick look at the integration tests this morning and noticed that JsonApiSource isn’t mapping ajax responses to additionalTransforms yet (the response json is currently added as an additional “transform")
I also noticed that you haven’t included replaceRecord but I’m not sure if you’re planning on supporting that.
Oliver Searle-Barnes
@opsb
Mar 10 2016 08:54
replaceKey is another one
Dan Gebhardt
@dgeb
Mar 10 2016 12:59
@opsb the JSONAPISource will definitely be supporting replaceRecord, but I'm not sure that support for replaceKeymakes sense
in order to support replaceKey, the key would have to be treated as an attribute
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:00
hmm
Dan Gebhardt
@dgeb
Mar 10 2016 13:00
the assumption here is that any replaceKey transform will originate from a remote source and won't need to be reapplied to it
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:01
yeah
it’s certainly an edge case, but in theory there could be two sources supplying remote keys
Dan Gebhardt
@dgeb
Mar 10 2016 13:02
yes, that's possible - I think sources need to treat those as noops
so we should support it by filtering it out instead of just erroring
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:02
that’s fine as long as they don’t have a cache
Dan Gebhardt
@dgeb
Mar 10 2016 13:03
right, well each source can decide
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:03
I guess that’s up to the Source to deal with though
they can just apply it locally
Dan Gebhardt
@dgeb
Mar 10 2016 13:03
:)
yep
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:03
cool, done
so additionalTransforms is a blocker for me at the moment
Dan Gebhardt
@dgeb
Mar 10 2016 13:03
ok, please tell me
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:03
I could probably fill in the gaps myself though if you don’t have time before this evening
Dan Gebhardt
@dgeb
Mar 10 2016 13:04
I'll be working on orbit today
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:04
great
Dan Gebhardt
@dgeb
Mar 10 2016 13:04
tell me what you need
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:04
ok, so currently the json from the ajax responses is being returned as additionalTransforms
they need to mapped to an empty array or new transforms
let me get a link, the code will make this obvious
Dan Gebhardt
@dgeb
Mar 10 2016 13:05
ok
The replaceAttribute method there is returning the results from source.ajax(source.resourceURL(type, id), 'PATCH', { data: json }); which is expected to be an array of additionalTransforms
I think it was removeRecord where I noticed this
presumably in that case it would just return an empty array
Dan Gebhardt
@dgeb
Mar 10 2016 13:09
right, or there's a default of an empty array
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:10
This methods in transforms return the json bodies from the ajax responses though
So you get that instead of the empty array default
Dan Gebhardt
@dgeb
Mar 10 2016 13:10
oh whoops
that's a mistake
ok, I can fix that easily
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:11
yeah, I figured you just hadn’t got to it yet
Dan Gebhardt
@dgeb
Mar 10 2016 13:11
good catch
I was worried there was an architectural problem
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:11
no, nothing major, just some dots that need joining
Dan Gebhardt
@dgeb
Mar 10 2016 13:11
:)
maybe these methods should always return the array including the original transform
to allow for ordering
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:13
you’d know better than me if that’s likely with json api :)
Dan Gebhardt
@dgeb
Mar 10 2016 13:13
obviously an internal concern
anyway, I've got some work to do to optimize all this
and implement replaceRecord
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:14
great
Dan Gebhardt
@dgeb
Mar 10 2016 13:14
so I'll see where it shakes out
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:14
ok, I’m going to be putting a fair push in this week I think
Dan Gebhardt
@dgeb
Mar 10 2016 13:14
nice :)
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:14
I’m keen to get liveQueries hooked up with ember-concurrency
I see there’s work underway to expose integration with Observables
Dan Gebhardt
@dgeb
Mar 10 2016 13:15
yes, machty is working on this
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:15
yeah
I think he’s paving the way for general RxJS adoption there
be interesting to see if it ends up being a dependency or not
Dan Gebhardt
@dgeb
Mar 10 2016 13:17
yeah, not sure how it will all shake out
e-c scratches an itch many folks didn't realize they had
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:18
yeah
it’s the automatic cancellation which people don’t seem to have thought about
Dan Gebhardt
@dgeb
Mar 10 2016 13:19
although there is a lot of talk about cancellable promises
which Domenic is working on as a standard
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:19
that would certainly be useful, for one off tasks
Dan Gebhardt
@dgeb
Mar 10 2016 13:19
I think there's a place for observables too
I will discuss more with everyone at EmberConf in a few weeks
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:21
yeah, I thought of a good example for orbit actually, with Observables we could allow Sources to have their own queues, but still allow external control in the case of an error i.e. on error the whole stream is thrown and can be caught anywhere up the chain.
Dan Gebhardt
@dgeb
Mar 10 2016 13:22
so we'd have to go full in with observables then
it's a compelling use case
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:22
yes, that’s the first concrete example I’ve been able to give
but I’m sure there are more
orbit is stream based after all
Dan Gebhardt
@dgeb
Mar 10 2016 13:24
essentially :)
well, one step at a time
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:24
here’s the Store/Source setup in the json api integration test at the moment https://github.com/opsb/orbit.js/blob/blocking-store/test/tests/integration/json-api-test.js#L81
the new updateFail hooks means that blocking behaviour is only used to throttle the queue
Dan Gebhardt
@dgeb
Mar 10 2016 13:27
so store.deny unwinds subsequent transforms too, right?
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:29
you mean transforms following the denied transform? (that arrived via jsonSource.transform())
Dan Gebhardt
@dgeb
Mar 10 2016 13:29
yes
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:29
we haven’t specified the behaviour for that yet
I think this matches how it was working previously anyway (correct me if I’m wrong)
Dan Gebhardt
@dgeb
Mar 10 2016 13:30
right, I wasn't sure if you had added a default behaviour
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:31
so yeah, at the moment it just deals with tranforms one at a time, whatever happens the following transforms will always be processed
we’re going to want some hooks around error handling though
Dan Gebhardt
@dgeb
Mar 10 2016 13:31
sure
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:31
we just need to decide what they are really
Dan Gebhardt
@dgeb
Mar 10 2016 13:33
so you've got #268 and #270 open
I guess that #268 includes #270 now
and that I could merge #270 any time
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:33
they’re both independent at the moment
but yeah, you can go ahead and merge 270
it just depends whether you want to hold off until you’ve figured out what changes are necessary for the QueryEvaluator
Dan Gebhardt
@dgeb
Mar 10 2016 13:34
gotcha
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:35
liveQueries are pretty isolated so there’s little danger of them going stale, aside from the QueryEvaluator changes obviously
Dan Gebhardt
@dgeb
Mar 10 2016 13:35
so I'll see if I work out the QueryEvaluator changes today and tomorrow and we can refactor 270 if needed
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:35
cool
yeah, while I’m keen to get liveQueries in, I’m more keen to get a release of the full stack done
Dan Gebhardt
@dgeb
Mar 10 2016 13:36
yes, way overdue
but we want to be pretty settled on the interfaces
which is why we haven't merged to master yet
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:37
true
Dan Gebhardt
@dgeb
Mar 10 2016 13:37
hopefully this weekend we can do so
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:37
yeah, that would be great
I’m planning on working on orbit over the weekend again
Dan Gebhardt
@dgeb
Mar 10 2016 13:38
great
Oliver Searle-Barnes
@opsb
Mar 10 2016 13:38
k, I’m heading out for lunch
Dan Gebhardt
@dgeb
Mar 10 2016 13:38
buen provecho!
Dan Gebhardt
@dgeb
Mar 10 2016 17:42
@opsb I believe this will unblock you orbitjs/orbit.js#273
This "Request" abstraction fits well with JSONAPISource
I'm going to proceed with coalescing ops into as few requests as possible.
Oliver Searle-Barnes
@opsb
Mar 10 2016 18:06
@dgeb that looks like it’ll take care of the additionalTransforms
I’m not sure I understand what the OperationToUpdateRequestMap is trying to achieve though, the requests seem very similar to the operations, is it just a semantics thing?
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:09
I went ahead and merged, just starting to look at the integration tests now
Dan Gebhardt
@dgeb
Mar 10 2016 19:09
ok, thanks
the requests are meant to align perfectly with possible ajax requests
to jsonapi endpoints
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:10
right, yeah, I grokked it after looking a bit closer
Dan Gebhardt
@dgeb
Mar 10 2016 19:11
it is a very subtle difference, but we'd get very subtle errors / misunderstandings if we didn't make it I think
and the same request concept can work for fetch as well
i.e. queries will need to be evaluated down to a fetch request
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:22
yeah, I can definitely see how it brings some clarity
it’s an explicit mapping step instead of being buried in the method that makes the call
as you say, it’s easier to spot issues
Dan Gebhardt
@dgeb
Mar 10 2016 19:24
:+1:
so let me know if you hit any more blockers
I hope to be working on this pretty exclusively for the next few days
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:26
will do, I’ll try and get through the transform tests today, then I’ll need jsonApiSource.fetch() :)
(I realise there’s a fair bit to do there)
Dan Gebhardt
@dgeb
Mar 10 2016 19:26
yeah, that's next on my list though :+1:
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:27
good good
with a bit of luck we should be in good shape by the weekend
Dan Gebhardt
@dgeb
Mar 10 2016 19:27
yep, fingers crossed
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:27
BTW, what’s your plan with ember-orbit for this release?
Dan Gebhardt
@dgeb
Mar 10 2016 19:28
I haven't focused on it in a while
I'd like this orbit release to just be a beta
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:28
I think aside from Transactions it should be pretty close
right
I’m right in thinking that your end goal is to use it with an ember app in the near future?
Dan Gebhardt
@dgeb
Mar 10 2016 19:29
definitely
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:29
ok
I might start looking at that over the weekend then
Dan Gebhardt
@dgeb
Mar 10 2016 19:30
that would be great :)
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:30
once the JSON API integration test is done
Dan Gebhardt
@dgeb
Mar 10 2016 19:30
right
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:30
seems to be next
I did a fair chunk of work on it over christmas so it’s still relatively fresh
Dan Gebhardt
@dgeb
Mar 10 2016 19:31
yeah, and it's working with livequeries now, right?
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:32
It works with them on the Cache, I need to join the dots up on the Store
Dan Gebhardt
@dgeb
Mar 10 2016 19:32
ok
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:32
although once they’re wired up in the orbit Store that should be pretty straight forward
Dan Gebhardt
@dgeb
Mar 10 2016 19:32
sounds good
we could even release with isolated transactions
such that the transaction cache needs to be "warmed"
(i.e. without the old failover behavior)
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:34
yeah, need to think about transactions some
Dan Gebhardt
@dgeb
Mar 10 2016 19:35
yeah, we can start simply and conservatively
until we're confident in an approach
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:36
I started looking at integrating the QueryBuilder into the Store interface, what do you think to https://github.com/opsb/orbit.js/blob/19726c3ecebc90ed8be8c2894111756070cf10ef/test/tests/orbit-common/unit/cache/live-queries-test.js#L80 ?
e.g.
    const liveQuery = cache.liveQuery(q =>
      q.recordsOfType('planet').filterAttributes({ name: 'Pluto' })
    );
queries would be the same
the builder style is a little different to the TransformBuilder, but it needs to be because the available methods change as you build the query
Dan Gebhardt
@dgeb
Mar 10 2016 19:38
sure
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:38
i.e. the TransformBuilder has a single term, the QueryBuilder has many
An alternative might be
    const liveQuery = cache
      .liveQuery
      .recordsOfType('planet')
      .filterAttributes({ name: 'Pluto' })
      .run();
Dan Gebhardt
@dgeb
Mar 10 2016 19:39
i.e. the TransformBuilder creates many transforms, one term each, while the QueryBuilder creates a single query from many terms
hmmm
I prefer the former
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:40
that’s what I thought
I think I do too
I’m not 100% yet though
Dan Gebhardt
@dgeb
Mar 10 2016 19:40
it's more consistent with transform builders
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:40
yeah
Dan Gebhardt
@dgeb
Mar 10 2016 19:41
plus it allows for objects to be passed instead of functions
which can be useful
we could pre-build queries
and pre-build transforms too
(that would be quite an optimization - not suggesting we try that now)
but it's possible this way
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:42
ha, yeah, I don’t see why not
it also leaves more room for experimentation, you can easily bring your own query builder
Dan Gebhardt
@dgeb
Mar 10 2016 19:43
right
:shipit:
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:43
done

So how about the following?

    const liveQuery = cache.liveQuery(q =>
      q.relatedRecords('planet', 'pluto', 'moons')
    );

vs

    const liveQuery = cache.liveQuery(q =>
      q.record('planet', 'pluto').relatedRecords('moons')
    );
There’s a long list of decisions to be made with the query builder, but I think most can be deferred for now. This one dictates the style to a degree though.
Dan Gebhardt
@dgeb
Mar 10 2016 19:47
hmmm
I think we need to consider this from the "built" form
I've been planning to distill queries down in jsonapi to "requests" with configuration options
and doing so from oql is a bit awkward
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:50
hmm, yes ok
where are the sticking points?
Dan Gebhardt
@dgeb
Mar 10 2016 19:51
things like sorting and filtering
which wrap the core query
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:51
right
Dan Gebhardt
@dgeb
Mar 10 2016 19:51
but then need to decorate it in request form
it's all possible
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:52
can’t those details drop away at the Source?
I mean, this is why fetch makes sense, it’s fetching the information that can be used to satisfy a query
But the actual query is performed in the Cache
Dan Gebhardt
@dgeb
Mar 10 2016 19:53
yes, but the filtering / sorting should also be applied at the server
so it needs to translate into a request
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:53
is there value in that if we throw away that information between the Source and the Store?
Dan Gebhardt
@dgeb
Mar 10 2016 19:54
well, getting the first 10 results depends on the order in which they're sorted
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:54
true
ok
sorting seems like it should map easily
I’m not sure what your thoughts are on filtering
That seems to be wide open in the json api spec
Dan Gebhardt
@dgeb
Mar 10 2016 19:56
yeah, I will implement to match jsonapi-resources' opinions
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:56
does it have opinions regarding filtering?
so there’s attribute based filtering
Dan Gebhardt
@dgeb
Mar 10 2016 19:56
yes, it's pretty basic
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:57
but I see you can map to named filters
so what are your thoughts here then, did you have a direction in mind?
Dan Gebhardt
@dgeb
Mar 10 2016 19:58
so I'm thinking we might want more of an AST for queries, that can then be processed by sources in different ways
we could generate this via oql or a query builder
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:59
I think we lose a lot if we don’t standardise that as much as possible
Dan Gebhardt
@dgeb
Mar 10 2016 19:59
I agree
Oliver Searle-Barnes
@opsb
Mar 10 2016 19:59
I’d like to see a core query set, perhaps with extensions for particular sources
although I think there will be a lot of overlap
Dan Gebhardt
@dgeb
Mar 10 2016 20:00
I think we could make the analog to record-aware transforms
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:00
for instance, named queries could be defined in the ember-model layer for the cache
with the server left to interpret as it needs to
interesting
Dan Gebhardt
@dgeb
Mar 10 2016 20:01
can you give me an example?
of named queries I mean
I think I know what you're getting at, but want to be sure
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:02
e.g. q.recordsOfType(‘planet’).namedFilter(‘visibleToTheNakedEye’)
Dan Gebhardt
@dgeb
Mar 10 2016 20:02
gotcha
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:03
that’s starting down the GraphQL route though
Dan Gebhardt
@dgeb
Mar 10 2016 20:03
so let me work up some examples to talk over
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:03
(I mention GraphQL only as a good reference for named queries)
Dan Gebhardt
@dgeb
Mar 10 2016 20:03
yeah
I figure it's fine for sources to have specific query operators
and other sources to simply fail
when they get those queries
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:05
I think that’s something we should avoid as much as possible
Dan Gebhardt
@dgeb
Mar 10 2016 20:05
we definitely need a base set, just like we have a base set of record-aware transform ops
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:05
but yes, in principle
sure
Dan Gebhardt
@dgeb
Mar 10 2016 20:06
I'm imagining some advanced queries done on the cache with custom JS functions involved
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:06
right
I think there’s a balancing act there
Dan Gebhardt
@dgeb
Mar 10 2016 20:06
indeed
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:07
We want to push people towards using queries which we can optimise with indexes etc.
Dan Gebhardt
@dgeb
Mar 10 2016 20:07
definitely
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:07
Otherwise you may as well just write a bit of javascript and pass in an array of records retrieved from the Cache
Dan Gebhardt
@dgeb
Mar 10 2016 20:08
sure - we should focus on standard queries - I agree
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:08
The value of the queries is in the optimisations and the convenience of having multiple Sources/Cache act accordingly
sure, ok, beating this one a bit ;)
Dan Gebhardt
@dgeb
Mar 10 2016 20:09
ha! I'm on your side, I swear ;)
Oliver Searle-Barnes
@opsb
Mar 10 2016 20:09
ha
well you do seem to appreciate standards so this seems up your alley ;)
ok, gotta run for dinner, will be around later
Dan Gebhardt
@dgeb
Mar 10 2016 20:10
sounds good - enjoy :)
Dan Gebhardt
@dgeb
Mar 10 2016 21:03
don't forget the ember community survey
hoping to have more than 1 user of ember-orbit listed :P
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:17
already done it I’m afraid :)
Dan Gebhardt
@dgeb
Mar 10 2016 21:18
well, hopefully ember-orbit can have a bit of a coming out party around that time :)
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:19
hmm, 21st, yeah, I think we should have something pretty decent by then
Dan Gebhardt
@dgeb
Mar 10 2016 21:19
:+1:
I guess I meant emberconf, which starts the 28th
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:20
even better
Dan Gebhardt
@dgeb
Mar 10 2016 21:20
yep :)
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:20
ok, that’s a good target
I’d like to think we can get everything together, liveQueries as well
That said, I’ve just come across a bug in the Cache :)
Dan Gebhardt
@dgeb
Mar 10 2016 21:21
oh no!
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:21
inverse links don’t seem to be getting added at the moment
jsut putting some tests together
Dan Gebhardt
@dgeb
Mar 10 2016 21:21
ok thanks
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:40
ah, false alarm, I thought the inverse relationships weren’t updating any more, but it was because I hadn’t defined the inverse in the schema...
Dan Gebhardt
@dgeb
Mar 10 2016 21:40
phew!
thought they were well tested
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:40
yeah, sorry about that one
I didn’t actually find a test for inverses
I’ll check it’s covered
Dan Gebhardt
@dgeb
Mar 10 2016 21:41
in the schema consistency processor
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:41
ah
Dan Gebhardt
@dgeb
Mar 10 2016 21:42
I'm writing a bunch of tests for the update requests now
really need to TDD the coalescing, etc
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:42
right, yeah
always relaxing when you get into the rhythm
Dan Gebhardt
@dgeb
Mar 10 2016 21:43
yep
crank up some tunes and go
much different than architectural planning
when I can listen to classical at most
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:44
yeah, much easier ;)
Dan Gebhardt
@dgeb
Mar 10 2016 21:44
yep :)
Oliver Searle-Barnes
@opsb
Mar 10 2016 21:51
yup, coverage looks pretty darn good there
Dan Gebhardt
@dgeb
Mar 10 2016 21:51
nice