These are chat archives for orbitjs/orbit.js

3rd
Sep 2015
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:34
@dgeb WRT per-source meta, how is it currently handled? Is the parallel structure something that we need to introduce or does it already exist?
Dan Gebhardt
@dgeb
Sep 03 2015 16:41
@opsb per-source meta is not handled yet
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:42
ok
Dan Gebhardt
@dgeb
Sep 03 2015 16:42
we would need to introduce the parallel structure
maybe retrieveMeta
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:42
how's it handled at the moment?
within the existing json structure?
haven't really paid much attention to it yet
Dan Gebhardt
@dgeb
Sep 03 2015 16:42
it's not - that's driving my work on this issue
we really need it
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:43
right, yeah, it would be really useful
I'm just starting to do some research on meteor's approach to optimistic ui and latency compensation, I think with some meta data and the new transaction work you've just done.
Dan Gebhardt
@dgeb
Sep 03 2015 16:45
cool - keep me posted :)
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:45
one of the things I think we'll need is the ability to rollback a transaction
I'm guessing that isn't implemented yet?
Dan Gebhardt
@dgeb
Sep 03 2015 16:46
well transactions already track their inverses
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:46
great
so it's just a case of adding a method to use those?
Dan Gebhardt
@dgeb
Sep 03 2015 16:49
so, typically transactions will be used to bundle together a transformation to be applied against the base source
and that transformation should either succeed or fail atomically
and when it succeeds, it will trigger a didTransform
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:51
yeah, so I suppose the basic idea with optimistic UI is that the transformation wouldn't be applied to the main source until the server returns (or the connectors resolve in orbit's case)
Dan Gebhardt
@dgeb
Sep 03 2015 16:52
with an optimistic UI, you're assuming success on the server
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:52
yeah
Dan Gebhardt
@dgeb
Sep 03 2015 16:52
so the UI can update immediately
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:52
the tricky bit is what happens if the server returns different information
Dan Gebhardt
@dgeb
Sep 03 2015 16:52
so if the server fails
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:52
in particular, what happens if the sequence at events at the server is different to the local client
so events that originate in the client will be earlier than in the sequence for that client than for other clients who receive the events via the server
if you end up with conflicts you need to roll back the transformations that have been eagerly applied and then roll forwards again based on the events received from the server
Dan Gebhardt
@dgeb
Sep 03 2015 16:54
right
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:54
that's the tricky bit :)
currently we're just using the naive approach of assuming no conflicts
which in practice has seemed fine actually
but obviously it would be good to get it done properly
Dan Gebhardt
@dgeb
Sep 03 2015 16:55
sure - but we need to track queues of transforms / results
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:55
yes
Dan Gebhardt
@dgeb
Sep 03 2015 16:56
and since transforms have IDs
we can roll them back based on the inverse operations
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:56
I think you'd need an additional source that only contained events from the server, with a blocking connector that waited until the server had caught up with local events
yeah
then when that source had caught up "something" would have to decide how to resolve any conflicts (the rollback then rollforwards)
that's as far as I've got so far anyway
Dan Gebhardt
@dgeb
Sep 03 2015 16:58
right
we are missing a few components
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:58
yeah
Dan Gebhardt
@dgeb
Sep 03 2015 16:58
but I think we have most of the primitives right now
Oliver Searle-Barnes
@opsb
Sep 03 2015 16:58
seems like there'd need to be something co-ordinating activities between connectors
but possibly the store could be responsible for that
Dan Gebhardt
@dgeb
Sep 03 2015 16:59
yeah, going to think on it more
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:00
I think the key component is the blocking connector between the source containing server events only and the store's main source
it would need to track the transformations that had been sent to the server and take care of blocking etc.
it might be something sitting between two connectors actually
one that tracks transformations that had been sent to the server, the other one that does the consensus
so yeah, bit rough at the moment
Dan Gebhardt
@dgeb
Sep 03 2015 17:05
yeah, a coordinator could allow sources to reset their logs as they progress and reach a new baseline
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:05
I'm pretty sure the logs will need to be ordered
but maybe if the co-ordinator tracked the sequence it wouldn't be necessary
we probably wouldn't want to keep all the transactions in each source (just the log) so it would have to be something external taking care of the rollback anyway
Dan Gebhardt
@dgeb
Sep 03 2015 17:07
yeah, order is important - something needs to track order to allow resets
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:07
yeah
a buffering connector could take care of that
this assumes you can match up transformations when they come back from the server, but I'll have to do that anyway
I'm also not clear on how you'd detect conflicts
Dan Gebhardt
@dgeb
Sep 03 2015 17:11
so the non-blocking connectors are throwing away the promise returned by the target - they could hold onto that for tracking - and then trigger an event if the promise fails
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:11
that works for failures, it doesn't take care of ordering issues
i.e. if you rename a task locally, and someone else renames it at the same time the sequence at the server will be different to the sequence arriving at the store's source (the local events will arrive faster)
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:16
this is particularly relevant in the offline scenario
It's the equivalent of git rebase really
Dan Gebhardt
@dgeb
Sep 03 2015 17:18
yeah, we can't generically provide offline support without a means to resolve conflicts
we have the tools to effectively do git reset --hard HEAD~3 for instance
and then apply new commits
but the coordination of that work doesn't really have a public API yet
and I want to provide this API
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:20
no, the reset is a little harder as well because we can't point to previous versions of the source's json
(we have to do a rollback)
but doable
Dan Gebhardt
@dgeb
Sep 03 2015 17:21
that's what I mean - we can rollback any transformation at a source that has a cache
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:21
yeah
Dan Gebhardt
@dgeb
Sep 03 2015 17:21
and we can rollback a whole queue of them even
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:21
yeah, as you say, all the primitives are there, just need a co-ordinator
Dan Gebhardt
@dgeb
Sep 03 2015 17:21
that's what I'm thinking
glad that you're looking at meteor and other solutions for inspiration
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:22
yeah, well, I started keeping an eye on rethinkdb issues, just to get a feel for what's going on
one of the things that came up is they're integrating with meteor
Dan Gebhardt
@dgeb
Sep 03 2015 17:22
ah
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:23
you might find this interesting https://www.meteor.com/ddp
the rethinkdb guys are toying with the idea of implementing it
Dan Gebhardt
@dgeb
Sep 03 2015 17:24
I should read up on it
and let them know about jsonapi operations
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:24
yeah
Dan Gebhardt
@dgeb
Sep 03 2015 17:24
seems to overlap
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:24
definitely
Dan Gebhardt
@dgeb
Sep 03 2015 17:24
meteor has serious NIH syndrome
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:25
ha
that's just JS ;)
Dan Gebhardt
@dgeb
Sep 03 2015 17:25
haha ... we're all a bit guilty
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:25
yeah, hopefully we'll get a little consolidation over the next couple of years
Dan Gebhardt
@dgeb
Sep 03 2015 17:25
yeah, some specs will win out
and the tooling will build up
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:26
gotta say, I'm still in two minds about whether glimmer was a good idea
given the existing options
Dan Gebhardt
@dgeb
Sep 03 2015 17:26
vs. using react?
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:26
yeah, or at least their vdom implementation
Dan Gebhardt
@dgeb
Sep 03 2015 17:26
hmmm
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:26
glimmer's a bit of a pain because it forces you to do everything in ember land
you can't manipulate the dom directly at all
it has the effect of creating an island
Dan Gebhardt
@dgeb
Sep 03 2015 17:27
in theory it allows for smarter, smaller dom updates
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:28
yeah, in practice that's the cost though, it's harder to leverage work done in other parts of the js community
for the UI anyway
Dan Gebhardt
@dgeb
Sep 03 2015 17:28
yeah, ember has some real interop issues
I've been convinced from the beginning to keep orbit plain js
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:28
the attitude seems to be that it's a shitstorm out there, let's make an island where everything is done properly
right
it certainly creates more opportunities
Dan Gebhardt
@dgeb
Sep 03 2015 17:29
yeah - not much has been realized yet
but I think we can solidify this very quickly
feeling better about orbit now than I have in a long time
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:30
I'm feeling a bit bitter about glimmer at the moment, was using jquery-ui-sortable before. But it doesn't work with glimmer so I'm currently bringing ember-sortable up to parity with it. Was not planned work...
yeah, I think making transformations immutable has made a big difference
Dan Gebhardt
@dgeb
Sep 03 2015 17:30
agreed
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:30
solves a lot of the issues I was looking at
Dan Gebhardt
@dgeb
Sep 03 2015 17:31
cool
I don't know if we need to go further with immutability and force it
with something like immutable.js
there's some overhead there
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:32
yeah, although potentially we could dedupe the json sources
Dan Gebhardt
@dgeb
Sep 03 2015 17:33
true! seems tricky, but interesting
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:33
i.e. you could use the transformation log to decide if two sources are equal
the latest transformation id is now effectively the revision identifier for a source
(except with sources using optimistic UI :) )
Dan Gebhardt
@dgeb
Sep 03 2015 17:34
sure :)
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:35
having said that, we might be able to have a revision number that is based on the sequence of transformations that gets updated after each transformation is applied
thinking about git again, he really nailed the model on that one
Dan Gebhardt
@dgeb
Sep 03 2015 17:36
yeah, it's great - and we all understand the model by now :)
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:36
yep
Dan Gebhardt
@dgeb
Sep 03 2015 17:37
I think we can drop caches from most sources by default now
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:37
right
Dan Gebhardt
@dgeb
Sep 03 2015 17:37
and stick with revisions / applied transforms
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:37
yeah, that's true, far more efficient
makes rollbacks easier ;)
Dan Gebhardt
@dgeb
Sep 03 2015 17:38
true :)
alright - I'm going to get to work on that new normalized form now
good chatting with you :)
Oliver Searle-Barnes
@opsb
Sep 03 2015 17:39
cool, yeah, things are moving forwards nicely!
Dan Gebhardt
@dgeb
Sep 03 2015 17:39
glad you agree!