Expensify Bedrock/Lobby

Marcelo Ruiz Camauër

@mruizcamauer_twitter

I want to have a network of devices that come and go at will (get turned off), in a local LAN. I want them to resynch so that any one of them can become the new Leader. I want to be able to make some nodes more likely to be leaders (since they will be full PC's), but I believe that is already a feature in Bedrock. My concern is what happens when only one device is left on, will it stop working due to lack of concensus? or can this be set so even if a single node is left it can continue working as the Leader?

David Barrett

@quinthar

hello @mruizcamauer_twitter! I think it should work just fine on a Pi; it doesn't do anything fancy.

As for synchronization, here's an overview of how it works: https://bedrockdb.com/synchronization.html

That said, any kind of synchronization requires a quorum of nodes to be online in order to commit anything -- otherwise you open yourself up to a "split brain" scenario (where something causes the cluster to break into two parts, each of which commits transactions that conflict with the other, "forking" the cluster).

Marcelo Ruiz Camauër

@mruizcamauer_twitter

What if you have a system where you may have one-N devices on at any time? The second one to come on should assume the first one is the leader and get an update from it. Can it be configured to work with only one device on?

Is Bedrock different than RQLite in some important feature? The plugins like cache and job queue are valuable, for instance.

Would bedrock be suitable for synchronizing a large number of instances? say 100-200, where the DB is quite simple however (low traffic and amount of data)

Marcelo Ruiz Camauër

@mruizcamauer_twitter

If you somehow suffer a split-LAN event, how do you determine which is the official and which the split, and how can you remerge them?

@quinthar how do you resolve this situation? how do you find the two halves?

David Barrett

@quinthar

If you somehow suffer a split-LAN event, how do you determine which is the official and which the split, and how can you remerge them?

The way we solve this is to configure every node to know every other node -- this way everyone knows how big the cluster is, and knows if it's able to talk with more than half of it. If a node can't talk with more than half of the cluster, then it will refuse to "stand up" to be the leader.

Would bedrock be suitable for synchronizing a large number of instances? say 100-200, where the DB is quite simple however (low traffic and amount of data)

I haven't tried that many, but I can't think of any reason why this wouldn't work. The one complication is, as mentioned above, the design requires that every node know every other node ahead of time -- new nodes currently can't be added or removed without restarting the cluster.

Is Bedrock different than RQLite in some important feature? The plugins like cache and job queue are valuable, for instance.

Honestly I don't know much about RQLite, sorry!

Marcelo Ruiz Camauër

@mruizcamauer_twitter

thanks for the replies!

Marcelo Ruiz Camauër

@mruizcamauer_twitter

I am trying to compile Bedrock on Mac OSX, and I get the following error: libstuff/libstuff.h:44:59: fatal error: pcrecpp.h: No such file or directory

include <pcrecpp.h> // sudo apt-get install libpcre++-dev

compilation terminated.
make: * No rule to make target .build/main.d', needed by.build/main.o'. Stop.

Marcelo Ruiz Camauër

@mruizcamauer_twitter

I installed the PCRE libraries, but still no go...

David Barrett

@quinthar

Ah yes, getting anything to compile in OSX is always a pain. Every time the OS updates, it seems to break all the paths

I have had so much trouble trying to basically do any development natively in OSX that I've just kind of given up. brew is always breaking, and I feel like I need to reinstall my entire OS whenever anything breaks.

The upshot is: yes, it compiles under OSX... if you can figure out how.

But it'll probably stop compiling when you upgrade the OS again.

Marcelo Ruiz Camauër

@mruizcamauer_twitter

any binaries already compiled? I need to take it for a spin to decide if we can use it in a really neat application...

David Barrett

@quinthar

Well where would you be deploying it? Probably not on OSX, right?

Marcelo Ruiz Camauër

@mruizcamauer_twitter

no, RPI

raspberry pi

if we can compile it for that, of course...

David Barrett

@quinthar

yes, any Linux environment it should compile much cleaner

I'd suggest just testing it on the environment that you intend to use it

Marcelo Ruiz Camauër

@mruizcamauer_twitter

One important thing is determining how quickly you can add new nodes, given a tiny db (3mb)... I imagine it would be a very quick synchronization, but if you had 50 clients, would the 51st be fast? would turning on all 50 be fast?

David Barrett

@quinthar

Well to clarify, the design isn't that you can just add new nodes automatically -- currently it requires that you reboot all the existing nodes to add one more. This is because every node needs to know how big the cluster is in order to know if it has quorum or not.

Additionally, it's not designed to "bootstrap" by downloading the database from another node; it instead downloads the most recent backup of the database (eg, from S3) and then synchronizes the most recent commits from a peer. (This is because each node only keeps the most recent N commits, not all commits back to the start of time.)

Marcelo Ruiz Camauër

@mruizcamauer_twitter

but copying the db from the leader would not be an option to getting the latest db?

"rebooting", how long does that take?

David Barrett

@quinthar

Well it depends on database size; it's as fast as it takes to download your database from S3.

Marcelo Ruiz Camauër

@mruizcamauer_twitter

!ok, it's all LAN, so it should be very fast1

managed to advance in compilation a little, it was missing paths to PCRE libraries (at least mine)... stuck now in another place

Undefined symbols for architecture x86_64:
"pcrecpp::RE::Init(std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pcrecpp::RE_Options const)", referenced from:
SParseHTTP(char const, unsigned long, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SString, STableComp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SString> > >&, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) in libstuff.a(libstuff.o)
BedrockPlugin_Jobs::_isValidSQLiteDateModifier(std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in libbedrock.a(Jobs.o)
BedrockPlugin_MySQL::onPortRecv(STCPManager::Socket*, SData&) in libbedrock.a(MySQL.o)
ld: symbol(s) not found for architecture x86_64

TAD Master

@teamtad_twitter

Hi. I am new to Bedrock -- I find the points for it quite justified, so am thinking of using it for my project. However, I have the following situation -- and I hope some guru here would let me know if that is possible. I want to give out an sqlite database to anyone who wants to read it from their part of the world -- So I want a master node with a set of readonly slave nodes.. That is the kind of distribution of the data I want. Basically, writing into the database would be only from the master (or some set of known nodes); but only reading should be possible on other nodes. Is this kind of distribution architecture possible with Bedrock? Thanks in advance.

David Barrett

@quinthar

@teamtad_twitter Yes, that's exactly how the design currently works! At any point in time there is one leader, and any number of followers. The leader is elected by a quorum of "full nodes" -- ie, anybody who you designate as a viable leader. But you can also designate any number of permanent followers, and thus they will not participate in the leader election (and will never become leaders). Rather, they will remain permanently read-only followers that escalate any attempt to write to the leader.

If the goal is to further constrain followers to not even escalate write commands to the leader, that isn't something currently supported out of the box, but would be pretty easy to do by creating a custom version of the DB plugin to make leaders just refuse writes escalated from permanent followers.

TAD Master

@teamtad_twitter

@quinthar Ah... thanks. Any sample code/configuration that I can use?

David Barrett

@quinthar

@teamtad_twitter Here's the code for the entire DB plugin -- as you can see it's pretty small: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp

Currently it has makes only a cursory attempt to determine if the query is a write before escalating it from the follower to the leader: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp#L59 This is because it's not currently designed to be deployed into an "uncontrolled" environment where one of the nodes is less trusted than others.

I would suggest inserting some check here to see if the query tried to write anything: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp#L130 And if it did, then check to see if the peer who escalated the command is a Permafollower, such as shown here: https://github.com/Expensify/Bedrock/blob/cf2c36027f569df277d3f97ea08ff8a917ad014a/sqlitecluster/SQLiteNode.cpp#L370

David Barrett

@quinthar

If the write command was escalated by a Permafollower, then just do something like STHROW("400 Unauthorized write query");or something like that.

David Barrett

@quinthar

As far as how to make the plugin itself, to start I'd suggest just cloning the Bedrock repo and copying DB.cpp to a new file SecureDB.cpp in /plugins -- that'll ensure it just compiles easy. Later when you have proven it out, you'll likely want to move your SecureDB.cpp plugin into your own private repo, but I'd suggest skipping that to start.

Once you make the SecureDB plugin, launch Bedrock with the -plugins SecureDB parameter to enable it (and notably, don't enable DB, as that doesn't have the security check you want)

As far as how to actually launch the various nodes of the cluster, a page on that is here: https://bedrockdb.com/multizone.html

It doesn't explicitly talk about how to configure a Permafollower, but it is as simple as making the -priority of the follower 0 in the follower's command line, and then adding ?Permafollower=true to the node definition in -peerList on all of the full nodes. So, like -peerList node1:9001,node2:9002?Permafollower=true would indicate that node2:9002 is a Permafollower. This is necessary to configure on the leader as well as the follower, because obviously the leader can't trust the follower to advertise its own permafollower status (given that you are using this to identify "untrusted" nodes).

That's a lot of information, but should give you some tips on how to get started. Good luck!

TAD Master

@teamtad_twitter

@quinthar Thanks a lot. That is fantastic. I will surely note this down somewhere and work on it

TAD Master

@teamtad_twitter

@quinthar Pardon me for asking a trivial question. I tried creating the SecureDB.cpp as suggested that is needed in this logic (I hope I located it in the right location)

if (SStartsWith(upperQuery, "INSERT ")) {
if(peer->params["Permafollower"] == "true") {
STHROW("400 Unauthorized");
} else
response["lastInsertRowID"] = SToStr(db.getLastInsertRowID());
}

... but sadly, am having trouble getting the code to understand "peer" variable

I tried this

Peer* peer = getPeerByID(command.initiatingPeerID);

(and I included some .h files at the top of the code)

This is in the BedrockPlugin_DB::processCommand(...) function but does not compile (that is how I came to know about "peer" variable) .

My C++ is quite rusty and I have not fully gone thru the code -- just kind of intuitively understanding what is to be done.

Where communities thrive

People

Repo info

Activity

include <pcrecpp.h> // sudo apt-get install libpcre++-dev