Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Marcelo Ruiz Camauër
    @mruizcamauer_twitter
    Undefined symbols for architecture x86_64:
    "pcrecpp::RE::Init(std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pcrecpp::RE_Options const)", referenced from:
    SParseHTTP(char const
    , unsigned long, std::
    cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SString, STableComp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SString> > >&, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) in libstuff.a(libstuff.o)
    BedrockPlugin_Jobs::_isValidSQLiteDateModifier(std::
    cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in libbedrock.a(Jobs.o)
    BedrockPlugin_MySQL::onPortRecv(STCPManager::Socket*, SData&) in libbedrock.a(MySQL.o)
    ld: symbol(s) not found for architecture x86_64
    TAD Master
    @teamtad_twitter
    Hi. I am new to Bedrock -- I find the points for it quite justified, so am thinking of using it for my project. However, I have the following situation -- and I hope some guru here would let me know if that is possible. I want to give out an sqlite database to anyone who wants to read it from their part of the world -- So I want a master node with a set of readonly slave nodes.. That is the kind of distribution of the data I want. Basically, writing into the database would be only from the master (or some set of known nodes); but only reading should be possible on other nodes. Is this kind of distribution architecture possible with Bedrock? Thanks in advance.
    David Barrett
    @quinthar
    @teamtad_twitter Yes, that's exactly how the design currently works! At any point in time there is one leader, and any number of followers. The leader is elected by a quorum of "full nodes" -- ie, anybody who you designate as a viable leader. But you can also designate any number of permanent followers, and thus they will not participate in the leader election (and will never become leaders). Rather, they will remain permanently read-only followers that escalate any attempt to write to the leader.
    If the goal is to further constrain followers to not even escalate write commands to the leader, that isn't something currently supported out of the box, but would be pretty easy to do by creating a custom version of the DB plugin to make leaders just refuse writes escalated from permanent followers.
    TAD Master
    @teamtad_twitter
    @quinthar Ah... thanks. Any sample code/configuration that I can use?
    David Barrett
    @quinthar
    @teamtad_twitter Here's the code for the entire DB plugin -- as you can see it's pretty small: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp
    Currently it has makes only a cursory attempt to determine if the query is a write before escalating it from the follower to the leader: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp#L59 This is because it's not currently designed to be deployed into an "uncontrolled" environment where one of the nodes is less trusted than others.
    I would suggest inserting some check here to see if the query tried to write anything: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp#L130 And if it did, then check to see if the peer who escalated the command is a Permafollower, such as shown here: https://github.com/Expensify/Bedrock/blob/cf2c36027f569df277d3f97ea08ff8a917ad014a/sqlitecluster/SQLiteNode.cpp#L370
    David Barrett
    @quinthar
    If the write command was escalated by a Permafollower, then just do something like STHROW("400 Unauthorized write query");or something like that.
    David Barrett
    @quinthar
    As far as how to make the plugin itself, to start I'd suggest just cloning the Bedrock repo and copying DB.cpp to a new file SecureDB.cpp in /plugins -- that'll ensure it just compiles easy. Later when you have proven it out, you'll likely want to move your SecureDB.cpp plugin into your own private repo, but I'd suggest skipping that to start.
    Once you make the SecureDB plugin, launch Bedrock with the -plugins SecureDB parameter to enable it (and notably, don't enable DB, as that doesn't have the security check you want)
    As far as how to actually launch the various nodes of the cluster, a page on that is here: https://bedrockdb.com/multizone.html
    It doesn't explicitly talk about how to configure a Permafollower, but it is as simple as making the -priority of the follower 0 in the follower's command line, and then adding ?Permafollower=true to the node definition in -peerList on all of the full nodes. So, like -peerList node1:9001,node2:9002?Permafollower=true would indicate that node2:9002 is a Permafollower. This is necessary to configure on the leader as well as the follower, because obviously the leader can't trust the follower to advertise its own permafollower status (given that you are using this to identify "untrusted" nodes).
    That's a lot of information, but should give you some tips on how to get started. Good luck!
    TAD Master
    @teamtad_twitter
    @quinthar Thanks a lot. That is fantastic. I will surely note this down somewhere and work on it
    TAD Master
    @teamtad_twitter

    @quinthar Pardon me for asking a trivial question. I tried creating the SecureDB.cpp as suggested that is needed in this logic (I hope I located it in the right location)

    if (SStartsWith(upperQuery, "INSERT ")) {
    if(peer->params["Permafollower"] == "true") {
    STHROW("400 Unauthorized");
    } else
    response["lastInsertRowID"] = SToStr(db.getLastInsertRowID());
    }

    ... but sadly, am having trouble getting the code to understand "peer" variable

    I tried this

    Peer* peer = getPeerByID(command.initiatingPeerID);

    (and I included some .h files at the top of the code)

    This is in the BedrockPlugin_DB::processCommand(...) function but does not compile (that is how I came to know about "peer" variable) .

    My C++ is quite rusty and I have not fully gone thru the code -- just kind of intuitively understanding what is to be done.

    Brian Lee
    @lkm1321_gitlab
    hey guys, thanks for sharing this awesome project. i'm just trying to understand what bedrock can/can't do, just wondering if anyone can help me out
    i'm looking for something that supports SQL read/write queries across multiple machines. it's okay that the sync is a bit delayed, but i need immediate read/write at least to the local DB.
    i tinkered a bit with rqlite, but realised that it only allows write to the leader/master node. in my application, i expect significant latency between machines, so this will be a problem..
    am i correct in understanding that bedrock allows both read/write to local db, and synchronisation happens in the background?
    also, would it be possible to use bedrock as a C/C++ library, similar to what sqlite3 provides?
    thanks in advance for help :)
    Jose V. Trigueros
    @jvtrigueros

    Hi all, I currently run a Discord bot with a SQLite backend database. It was working fine until I enabled sharding, meaning there are multiple threads writing/reading from the same SQLite database. I've enabled WAL mode however I still get SQLITE_BUSY error codes.

    I stumbled upon Bedrock, so I figured I could give it a shot. I was able to build and run Bedrock successfully, pointing to my existing SQLite file, however, because there's no Java SDK for Bedrock I decided to give the MySQL plugin a shot. This is where the horror story starts, I'm only able to make SQL queries by connecting to Bedrock via the MySQL CLI client, but not if I'm connected via my bot using the ORM.

    Am I out of luck here?

    Paul Bergeron
    @dinedal

    I hope this channel can help me: I just created Expensify/Bedrock#788

    The tests pass and it is running, but it doesn't seem to matter what I send on 8888, I get no response. Any advice?

    $ nc localhost 8888
    SELECT 1;
    Query: SELECT 1;
    Query:
    SELECT 1;

    All with no response

    Paul Bergeron
    @dinedal
    Additionally, the mysql client always responds with ERROR 1045 (28000): Access denied
    David Barrett
    @quinthar
    @teamtad_twitter Hey sorry for the slow response, can you share a link to your code, or copy/paste the relevant section here?
    Lmk the actual compilation errors
    David Barrett
    @quinthar
    @lkm1321_gitlab Sorry for the delay -- yes, all writes are (currently) escalated to the leader. But all reads are done locally. But yes, there's a C++ "Plugin" interface. Here's a really basic example: https://github.com/Expensify/Bedrock/blob/master/plugins/DB.cpp
    @dinedal Can you check /var/log/syslog to see if any errors are being output? Also, what is the exact command line you are running?
    Zachary Whitley
    @zacharywhitley
    Hi, I just started checking out bedrock and wanted to let you know that I didn't see any docs for installing on CentOS 7 but I did manage to get it built :)
    Zachary Whitley
    @zacharywhitley
    Here's a gist if you're interested in adding it. I haven't had a chance to go over it closely but it worked for me https://gist.github.com/zacharywhitley/3d159d3c78a49b54e2fbbe51b79b128a
    Thomas
    @symgryph
    Anyone got this working on alpine?
    I was trying to get to work on embedded system
    g++-9 -o bedrock .build/main.o -Lmbedtls/library -L/Bedrock -rdynamic -lbedrock -lstuff -ldl -lpcrecpp -lpthread -lmbedtls -lmbedx509 -lmbedcrypto -lz
    /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /Bedrock/libstuff.a(SLog.o): in function SLogStackTrace()': /Bedrock/libstuff/SLog.cpp:10: undefined reference tobacktrace'
    /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /Bedrock/libstuff.a(libstuff.o): in function SGetCallstack[abi:cxx11](int, void* const*)': /Bedrock/libstuff/libstuff.cpp:107: undefined reference tobacktrace_symbols'
    /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /Bedrock/libstuff.a(libstuff.o): in function SException::SException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SString, STableComp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SString> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': /Bedrock/libstuff/libstuff.cpp:95: undefined reference tobacktrace'
    /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /Bedrock/libstuff.a(SSignal.o): in function _SSignal_StackTrace(int, siginfo_t*, void*)': /Bedrock/libstuff/SSignal.cpp:151: undefined reference tobacktrace'
    /usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /Bedrock/libstuff/SSignal.cpp:157: undefined reference to `backtrace_symbols_fd'
    collect2: error: ld returned 1 exit status
    make: * [Makefile:114: bedrock] Error 1
    just testing or are we SOL for musl?
    David Barrett
    @quinthar_twitter
    Hm, I haven't done any embedded toolchain stuff in quite a while, but no effort has been made to make Bedrock compile there. I'm sure it's doable... just not done. Sorry!
    János Veres
    @jveres
    Hi! I created a super minimal Docker image for Bedrock, it's less than 6MB. If anyone's interested it's available at https://hub.docker.com/r/jveres/bedrock-xsim.
    trancephorm
    @trancephorm
    Hello everyone, been researching lately what decentralized database solutions exist and I must say Bedrock seems pretty exciting to me, still I cannot fully grasp how it relates to decentralization property, better said how much it is censorship proof? I'm especially interested how it relates to Chromia and Wip2p, what happens if one node is attacked and start malicious edits or deleting of data?
    David Barrett
    @quinthar_twitter
    Hi @trancephorm nice to meet you! I don't think it's really designed for that use case. BedrockDB is designed for all nodes to operate on servers that you control, for queries you write. It's not a database opened up to arbitrary parties. Now, with plugin you could add an authentication layer and only expose stored procedures (rather than enable arbitrary queries) -- this is what we do for Expensify. But still, all of the nodes of the cluster need to be running on your own servers, as it can't somehow differentiate between "good" and "bad" queries.
    trancephorm
    @trancephorm
    @quinthar_twitter Thanks for this clarification. And what exactly is Expensify? If the nodes are dispensed thought the whole Internet, would the client connect to nearest node? Let's say I trust my nodes, and if there are many of them, as there is no single point of failure, may we say this is pretty much censorship resistant system? Actually Bedrock may be the best option for me, because of such properties.
    David Barrett
    @quinthar_twitter
    Hm, BedrockDB is basically a replacement for MySQL: anything you'd consider using MySQL for, you can use Bedrock for. But you wouldn't use MySQL or Bedrock for, say, a distributed database where you don't control every node. If you are afraid of one of the nodes being compromised this isn't a good platform, because any node can see and modify anything. On the other hand, if you are merely concerned about something being knocked offline and becoming inaccessible, then yes, Bedrock could be perfect because it's very fault tolerant
    (You can check out https://expensify.com -- it's a mobile app for scanning receipts and getting reimbursed)
    trancephorm
    @trancephorm
    So there's no some automatism in discovering the nearest node, I guess some DNS balancing is how load is balanced over the nodes, which is not considering what node is the closest?
    David Barrett
    @quinthar_twitter
    Well that would happen at a different layer -- you'd use something like IP anycast for that, which would cause DNS to resolve to the nearest server.
    trancephorm
    @trancephorm
    What attracted me to Bedrock is SQL compatibility, yet I think solutions like https://wip2p.eth.link/ are more suitable for my use case, but I don't have idea how I would make relational database work on key-value based "database", because abstraction layer is so spartan I wouldn't even call it a database.... Anyways, thanks for your responses!
    umbrellerde
    @umbrellerde

    Hello there, I am just starting with Bedrock Jobs and have some questions, I hope this is the right place?
    I want my consumers to wait up to 30s for any Job thats with "hello.", so I send this request:

    GetJob
    name: hello.*
    connection: wait
    timeout: 30000

    But this command immediately returns a "404 No job found". Shouldn't it wait 30s before returning?

    Joe Mordica
    @jmordica
    Hi there! A couple of questions if you don’t mind. I recently came across bedrock via a TechCrunch article and it’s very intriguing. We run our own telephony platform in a multi cluster GKE env (multiple regions) and this seems to be a good fit.
    1. can bedrockdb assign priority value dynamically?
    1. What needs to happen if nodes need to be added to the cluster at a later point in time?
    1. Is the data for designed to be ephemeral? Or should you always keep the data dir as fresh as possible and not rely on backups when/if a reboot of a node happens?