Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Nov 29 14:29
    gsvarovsky review_requested #120
  • Nov 29 14:11
    dependabot[bot] closed #119
  • Nov 29 14:11
    dependabot[bot] edited #119
  • Nov 29 14:11
    dependabot[bot] commented #119
  • Nov 29 14:06
    vercel[bot] commented #120
  • Nov 29 14:05
    dependabot[bot] edited #119
  • Nov 29 14:05
    vercel[bot] commented #120
  • Nov 29 14:05
    gsvarovsky synchronize #120
  • Nov 29 10:42
    vercel[bot] commented #120
  • Nov 29 10:42
    gsvarovsky opened #120
  • Nov 22 06:45
    vercel[bot] commented #102
  • Nov 22 06:45
    vercel[bot] commented #102
  • Nov 22 06:45
    dependabot[bot] labeled #102
  • Nov 22 06:45
    dependabot[bot] opened #102
  • Nov 22 05:43
    vercel[bot] commented #119
  • Nov 22 05:42
    vercel[bot] commented #119
  • Nov 22 05:42
    dependabot[bot] labeled #119
  • Nov 22 05:42
    dependabot[bot] opened #119
  • Nov 21 08:08
    gsvarovsky edited #118
  • Nov 21 08:02
    gsvarovsky opened #118
Angus McAllister
@Angus-McAllister
OK, we could somewhat arbitrarily categorise developers who would find m-ld useful thus:
  1. Completely unversed in linked data and CRDTs;
  2. Somewhat knowledgeable about one or both concepts; and
  3. Familiar with both.
1 reply
The category you have in mind here is the first, from what you've said. Generally speaking, the best approach for folks at that stage of familiarity is to be pretty prescriptive about how to use it.
Michiel de Jong
@michielbdejong
But we also need to think about software development teams. I myself may be somewhere at 1.5 on that scale, but I still want to be conservative about following common practices for two reasons:
1) not alienating other (potential) contributors
2) not reinventing the wheel or not risking to have to do so: developing an app already costs brain power, if I also do it in an uncommon stack or architecture, I'm adding burden that will slow my app development down
7 replies
Angus McAllister
@Angus-McAllister
Absolutely; we're very much aware of that, and don't want to make it unnecessarily difficult for people to use m-ld in their app development. The intention is actually to help speed it up, through them no longer having to worry about concurrency, replicated persistence, etc.
In that vein, since most applications use a database, that brings a development overhead of its own, in that the database has to be designed, defined, and configured, along with Object-Resource Mapping in the code, which can get pretty hairy, and introduce maintenance difficulties of its own. Not to mention the operational overheads....:-)
Michiel de Jong
@michielbdejong
When I use a common software architecture like MVC on top of ORM, I know that if I run into questions about anything, other people will have developed solutions (I can expect something like https://guides.rubyonrails.org/active_record_migrations.html will also exist for a framework like Symfony or Laravel. And even whole reusable application modules like "user session management" or "chat" can be at my disposal. There is a certain safety in walking the beaten path. When choosing something like Solid or remoteStorage or m-ld as the basis for my app, I am taking on a risk of unknown unsolved problems. What can we do to reduce that risk.
The choice for m-ld is independent of the choice for json-rql right? Or would you recommend them in connection to each other?
3 replies
Angus McAllister
@Angus-McAllister
That's an entirely fair question; don't get me wrong! Novel technologies like m-ld are a bit like exothermic reactions, which produce a net release of energy, but require initial energy input to get going. We need to reduce that up-front investment of effort!
Michiel de Jong
@michielbdejong
What if only part of my data needs to be synced? How would I set that up in m-ld?
7 replies
Can the same piece of data belong to multiple m-ld domains?
4 replies
Thinking for instance of the Federated Timesheets project which we both know well, does Timeld use one m-ld domain per 'timesheet'?
3 replies
Or one per 'worker'?
Michiel de Jong
@michielbdejong
I realise the main use case you had in mind when designing m-ld is one logged-in user (or one group of fully trusted collaborators), syncing data across the various user devices and to one or more cloud servers for persistence maybe. But I think it can also be very useful to use m-ld for syncing part of that user's data to other (not fully trusted) parties.
1 reply
For that, you would want to be able to create a sync domain ad-hoc (for instance when a user clicks 'share'). In an issue tracker, I may want to allow you to add issues to my tracker, and close issues that you created, but not to close issues created by others. That sort of business logic is usually in the controller of an MVC app, right? But if I give you access to send changes directly to my data store without going through my controller, I need to make a paradigm shift about where that access-checker code would live.
14 replies
Michiel de Jong
@michielbdejong
I guess I would want to move this code out of my controller and put it into some m-ld callback somewhere?
Michiel de Jong
@michielbdejong
Going back to Prejournal, it currently has a pretty loose architecture, consisting of two things: a database schema and a collection of commands. Each command is a function from array-of-strings to array-of-strings and it does its own access control & filtering based on the credentials supplied and the server config.
Michiel de Jong
@michielbdejong
Anticipating that I would sometimes want to share all movements from a given worker, and at other times will want to share all movements to a given project, it would make sense to create one domain per worker-project combination, right? And then to make them findable, maybe have some indexes of projects-per-worker and workers-per-project?
Michiel de Jong
@michielbdejong
Hm, that quickly becomes complex maybe.
What if I use just one domain, and then filter out which read/write actions are allowed with which credentials using m-ld/m-ld-js#94 ?
George Svarovsky
@gsvarovsky
Yes. However you still have the question of what sets of data you want to replicate, given that a domain is replicated in full.
However
Going back to your opening question: I'm not yet convinced that partitioning the data up into m-ld domains that are the only way to access the data, is necessarily the right approach for a full bug-tracking system, today. It might be in future when we have a nice pattern for m-ld/m-ld-spec#25. Maybe in this project we can research that.
Michiel de Jong
@michielbdejong
Michiel de Jong
@michielbdejong
I could imagine having 7 m-ld domains, one for each access area. In domain 1, store the list of bugs with their status. A and B can write there, C can only read. In domains 2A, 2B and 2C, store A's, B's and C's draft-state comments respectively. In domains 3A, 3B and 3C, store A's, B's and C's final-state comments respectively.
The tricky thing is that a comment would move from e.g. 2A to 3A when you mark it as final, which is a bit awkward.
It would be nicer to keep all A's comments in 2A (not only draft but also final ones), and then have a proxy that reads 2A and publishes 3A as a read-only filtered view on 2A.
I guess that's what's called a View in MVC :)
Michiel de Jong
@michielbdejong
But for now I guess one domain per access zone would be the way to go, right?
Angus McAllister
@Angus-McAllister
The partitioning strategy for domains would be in a different dimension, namely one domain per ticket. There would be a separate domain listing the ticket identifiers.
At first glance, that might seem like a gargantuan number of domains, but in practice, the overhead is likely to be very low, and has the added benefit of addressing the access control requirements.
Michiel de Jong
@michielbdejong
But how do you then prevent e.g. company C from reading the draft comments of company B?
Angus McAllister
@Angus-McAllister
So here, a future version of m-ld that supports granular read access control would meet that requirement, but the interim approach would be to stand up an additional temporary domain for the ticket containing just the draft comment data, shared amongst clients within a single system, and then when finalised, the data would be written to the main m-ld domain for the ticket, and the temporary domain jettisoned.
Angus McAllister
@Angus-McAllister
The draft data would only be visible to users of the system in which the ticket originated, since the temporary domain itself is not shared with other systems.
Michiel de Jong
@michielbdejong
Interesting, thanks! In my strawman requirements, draft comments are only hidden for C (maybe draft is the wrong word, sorry), so then maybe there could be 1 domain for "drafts A + B" that does not replicate to C, and then 1 domain for all other data about the bug.
1 reply
That solves who can see what (in the sense that, if organisation C does not have read access to something, then that data should also not be replicated to a m-ld node that is under organisation C's control)
1 reply
Marc Laporte
@marclaporte_gitlab
Tiki v25 now uses TUI editor which is a WYSIWYG editor which is Markdown-centric and not HTML-centric. This is what they say about future realtime capabilities: nhn/tui.editor#2336
2 replies
Marc Laporte
@marclaporte_gitlab
@Angus-McAllister @gsvarovsky As a follow up to the discussion todat
today
I encourage the goal of having a m-ld package on Packagist. As an example, we add the Cypht webmail client and we got over 50k installs: jasonmunro/cypht#311
Marc Laporte
@marclaporte_gitlab
There is not a lot for realtime: https://packagist.org/search/?tags=realtime
George Svarovsky
@gsvarovsky

Interesting... we're thinking that the first step for PHP support would involve a 'm-ld service' process, which communicates with the PHP via the web server https://github.com/m-ld/m-ld-spec/wiki/PHP-app

There would still be a lot of value in having a PHP library for integrating with it, which could then later use a fully-native PHP m-ld engine.

Marc Laporte
@marclaporte_gitlab
Sounds good to me
Danny Ayers
@danja
I'm mighty interested. But chatting with Bergi (Thomas Bergwinkle) earlier, he mentioned the IDs for bnodes thing. Do you have anything in place for that?
also...I really am just starting looking... say I want to make a thing that talks over SPARQL - how do I flip between endpoint X and the store between a m-ld setup?
(bergi probably has an RDF diff in JS, worth talking to)
George Svarovsky
@gsvarovsky

Hi Danny, thanks for dropping by, and for your interest.

he mentioned the IDs for bnodes thing. Do you have anything in place for that?

m-ld's API uses JSON-LD for writing data. If a JSON-LD object does not have an @id, it's given a generated one by the engine (we "skolemize" it). This means that blank nodes never appear in the data.

say I want to make a thing that talks over SPARQL - how do I flip between endpoint X and the store between a m-ld setup?

There are a couple of things you could mean here. A m-ld "clone" (a physical instance of the distributed "domain") is intended to be embedded in the application code, behaving more like a model, in the MVC sense, than like a database. Engines like the Javascript engine can provide local storage of the model for offline use and for data safety.

So, it's normal for an app to talk direct to its embedded clone, but you could also wrap up a clone with a remote API, and you'd have something that looks like a triplestore. That remote API could accept SPARQL; in fact it could be a full LDP. (In the case of the Javascript engine, it helps that it accepts SPARQL algebra natively.) However this approach means you can now have multiple remote clients, who might disagree, so you'd need to have some kind of locking, so you don't get to use m-ld's native multiplayer convergence algorithm.

So it might be better to have the switch in your client app, which decides whether the data it wants to look at is in a m-ld domain, and if so, clones it locally. It can still use SPARQL, and it could even do distributed queries across both the m-ld domain and remote conventional triplestores, using Comunica.

Finally, you might mean that you want to replace how the m-ld engine itself persists data internally. We're considering adding an extension API to do that – for example, for server-based clones that want to persist to a cloud-managed triplestore. This is not entirely straightforward because a m-ld clone has to atomically persist not just the triples but a bunch of control metadata too.

Sorry, huge download of info! I'd be interested to learn what your proposed app looks like, happy to discuss further. (BTW if you want to go more long-form you're welcome to post in the Discussion forum).

Danny Ayers
@danja
thanks George!
Danny Ayers
@danja

re. generating @id - ok, makes sense. A while back I worked for Talis and their platform/store took essentially the same approach, seemed to work :)

re. clones/domains/endpoints, yeah...I haven't fully absorbed the clone model yet, I guess I should have a play. I hadn't seen Comunica before - nice!

re. proposed app - I need to learn typescript/angular for some work-work so I was thinking of reworking some of the bits over here : https://github.com/danja/HKMS#components - mostly built around the idea of browser-based views querying a remote SPARQL server

  • but I also want to play with local-first kind of setups, so incorporating m-ld somehow.
    right now I need a decent link bookmarking app so that's gonna be my first target
George Svarovsky
@gsvarovsky
Sounds great, thanks Danny. We're aware of the steep initial curve getting started with m-ld, so we're more than happy to help, and any feedback you have about docs/examples etc. are most welcome. We're planning an overhaul of those, and some basic infrastructure bits to save on setup.
Link bookmarking sounds nice. Kind of amazing it continues to be a poorly-solved problem on today's web!
Michiel de Jong
@michielbdejong