by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    automenta
    @automenta
    https://github.com/automenta/netjs/blob/master/plugin/wikidata/update.sh this can be used to extract the data properties, if we just want to add to the ontology
    brb
    Marcel Klehr
    @marcelklehr
    ah, yes. the CRDT people flatten and re-balance their trees...
    automenta
    @automenta
    starting
    {"id":"083B9B869DEC5BF69730CB87","clock":{"083B9B869DEC5BF69730CB87":1401393919888.002}}
    "SYNC"
    [["a",1],1401393919888,"083B9B869DEC5BF69730CB87"]
    [["b",1],1401393919888.001,"083B9B869DEC5BF69730CB87"]
    [["a",2],1401393919888.002,"083B9B869DEC5BF69730CB87"]
        var Model = require('scuttlebutt/model');
    
        var m = new Model();
    
        debugScuttle(m);
    
        m.set('a', 1);
        m.set('b', 1);
        m.set('a', 2);
    function debugScuttle(m) {
    var debug = m.createStream({writable: false, sendClock: true});
    debug.pipe(process.stdout);
    }
    automenta
    @automenta
    automenta/netjs@8c48249
    p2p experiments with scuttlebutt piped through smokestack
    scuttlebuck doesnt seem to work to synchronize a data structure on a broadcast pipe. so what i tried was using a separate incoming and outgoing Model (key/value index) for each node
    this way a node is free to decide if it wants to replace its data with incoming data. but i guess this change gets broadcast too.. so im not sure if its efficient
    but node -> node streams should work correctly to synchronize data. but in that case, if a node has multiple neighbors, it will probably end up having to create a separate buffer for each neighbor
    automenta
    @automenta
    https://www.npmjs.org/package/n2n uses smokesignal, seems like a simple wrapper of it
    Marcel Klehr
    @marcelklehr
    Ah, I think n2n's author originally intended to make it a wrapper but decided to drop smokesignal afterwards
    yes, that's how you'd do it, pipe your scuttlebut to each peer directly, You would use the broadcast for publicizing new objects or sth like that
    automenta
    @automenta
    i got an example working that uses scuttlebutt with broadcast
    using the seprate inbox/outbox thing
    i copied smokesignal into a folder in the netention source to do some direct experiments, but if nothing changes ill switch back to the package version
    i tried disabling base64 and it still works, base64 should be less efficient when binary can transmit
    was looking at PSON https://github.com/dcodeIO/PSON
    and JSONStream
    i got JSONStream working in the following stream: MongoDB -> Netention object filtering -> Express request response
    so that should eliminate needing to store an entire query result in memory at any given time
    it could be separated from the netention stuff and used as a smokesignal example
    it creates N peers, each connected to the previous one. alternate topologies could be programmed
    alternate seed topologies to be specific. i dont know how these nodes will choose which other nodes to connect to
    on the first node i set a pipe from broadcast to stdout to see what's being transmitted
    using that i was able to eliminate some duplicate feedback from scuttlebutt by setting the incoming stream only half duplex
    automenta
    @automenta
    so the result is that the inbox should remain synchronized with the sum of all other peers outbox contents
    ie. inbox should be much larger than outbox
    but peer -> peer direct synchronization will be preferable when possible
    Marcel Klehr
    @marcelklehr
    yea.
    automenta
    @automenta
    i needed to start each node with a delay. when i oriignaly tried smokesignal i created two peers simultaneously and they connected but immediately disconnected. then 10s later reconnect
    maybe there's a way to allow nodes to remain connected even if they are created simultaneously
    Marcel Klehr
    @marcelklehr
    peer -> peer should actually always be possible, since nodes can find peers automagically ;)
    automenta
    @automenta
    cool
    Marcel Klehr
    @marcelklehr
    I feel like smokesignal needs some love, in general..
    automenta
    @automenta
    are u familiar with PSON or any of these other JSON compressors?
    my code as well
    Marcel Klehr
    @marcelklehr
    just checked out PSON, seems very useful for smokesignal
    automenta
    @automenta
    PSON can compress JSON to a Buffer
    and use a dictionary that adapts to elements used by the JSON
    (more efficient if the dictionary is known ahead of time)
    Marcel Klehr
    @marcelklehr
    bbbiab
    automenta
    @automenta
    PSON might be better applied in the scuttlebutt protocol because scuttlebutt emits the raw stream, not the objects that PSON would be encoding/decoding. but smokesignal could use PSON too in a method specifically for sending objects