Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Mark Tomlin
    @Dygear
    @andrewchambers Oh my god! This is so helpful! Seeing how it actually works is SO HELPFUL! Thank you so much!
    This is also really helpful https://github.com/andrewchambers/bupstash/blob/59cbd27c0cb949331abea077e0e08108ee0fb6d1/src/xid.rs#L57
    andrewchambers
    @andrewchambers
    no problem
    my stuff might not even be doing everything in the nicest way, but it works haha
    Mark Tomlin
    @Dygear
    This is already way better than what I've managed to come up with. I can at least start to build examples off of this for documentation by example purposes.
    Juhan Oskar Hennoste
    @FeldrinH
    I need to access an sqlite database from one thread for writing and at the same time from multiple other threads for reading. Is there some general recommendation / documentation on how that should be done?
    Thom Chiovoloni
    @thomcc
    yep:
    1. use wal mode
    2. one connection per thread
    that's actually the best possible case — if you do your threading like that you should never have reads/writes fail due to failing to get a lock in time
    (like you do if you need more than 1 writer)
    how long are read/write transactions likely to be held open for?
    the two concerns are:
    • if you might have writer starvation then you may need to worry about checkpointing
    • if writes are exceptionally rare then wal mode is possibly overkill
    (wal mode is enabled using PRAGMA journal_mode=WAL - https://sqlite.org/wal.html )
    Juhan Oskar Hennoste
    @FeldrinH
    writes are probably going to be much more common than reads, but the writes should be fast (1 row) and the reads should be longer (100-1000 rows)
    it's a mainly for logging incoming data, that is constantly coming in 1 row at a time and at some points when somebody wants to read the data, they will need to query a sizable amount of the newer rows
    Thom Chiovoloni
    @thomcc
    then yes, wal is a good fit
    Juhan Oskar Hennoste
    @FeldrinH
    thank you, this was very helpful
    Thom Chiovoloni
    @thomcc
    usually when people ask that question they have use cases with less clear answers
    one thing worth noting is checkpointing though. your use case shouldn't cause it to be an issue, but https://sqlite.org/wal.html#ckpt
    essentially sqlite under wal will write operations to the write-ahead-log, and periodically flush them to the db
    it does this by default and the defaults are fine, but it means you have this extra mydb.sqlite-wal file next to your db (if you named your db mydb.sqlite)
    don't delete this file — it's part of the db. if you end up noticing that it gets big, it means you probably have readers holding long transactions. in practice you proabbly dont need to worry about it, but i figured i'd mention it
    also, you'll have mydb.sqlite-shm sometimes. it's been a while, but iirc this one gets cleaned up if all processes exit cleanly. either way, don't delete it either
    so you have 3 files for the db. if that's a problem, then don't use wal
    Thom Chiovoloni
    @thomcc
    but wal is nice because readers don't block a writer and a writer doesn't block readers
    (open read transactions do block checkpointing though, but it's only really an issue if your read transactions are open for long periods of times)
    you may also want to set PRAGMA synchronous=NORMAL, which should have much better performance for writes under wal.
    Thom Chiovoloni
    @thomcc
    the downside is there's a brief point between when the write transaction is committed during which a power loss will lose the write transaction, even though it was committed. that said,
    • this cannot corrupt your db. it behaves equivalently to if you did a rollback before commiting the transaction
    • it takes a power loss for it to occur, e.g. the OS has to turn off before syncing the buffer. /panics/error/crashes (even segfaults or whatever)/etc will never trigger it
    i always turn it on but if you need to absolutely guarantee that a commited transaction is definitely there, you shouldn't
    also, be sure to use features=["bundled"] or some other guarantee of a modern sqlite
    old sqlite is a lot slwoer
    andrewchambers
    @andrewchambers
    @FeldrinH to avoid busy errors you should also use
    begin immedidiate on the writer
    this will grab the write lock eagerly and prevent sqlite3 from hitting certain cases where it cannot upgrade the lock and aborts the transaction
    Don
    @AwesomeIbex

    Hi, anyone know why this might happen?:

       = note: /usr/bin/ld: /build/wormhole/target/x86_64-unknown-linux-musl/debug/deps/liblibsqlite3_sys-ed0d2a6bd5718b21.rlib(sqlite3.o): unable to initialize decompress status for section .debug_info
          /usr/bin/ld: /build/wormhole/target/x86_64-unknown-linux-musl/debug/deps/liblibsqlite3_sys-ed0d2a6bd5718b21.rlib(sqlite3.o): unable to initialize decompress status for section .debug_info
          /build/wormhole/target/x86_64-unknown-linux-musl/debug/deps/liblibsqlite3_sys-ed0d2a6bd5718b21.rlib: error adding symbols: File format not recognized

    I'm building for a couple of architectures and this only happens in CI. It seems like the libsqlite3 that gets built in CI is probably from a diff arch?
    Tried cleaning and whatnot. I'm also forcing the linker in my cargo conf

    andrewchambers
    @andrewchambers
    @AwesomeIbex could be a problem with lto
    just a guess
    e.g. the rust files are lto enabled and the sqlite3 ones are not
    something like that
    Thom Chiovoloni
    @thomcc
    do you not have zlib installed? it sounds like something is producing an elf with a compressed debuginfo section, but it cant decompress (which uses zlib). it might be invalid also.
    Don
    @AwesomeIbex
    Interesting, i'll take a look today
    Don
    @AwesomeIbex

    So i'm manually installing zlib to make sure (just incase).

    Also i found we are using lto and for one architecture we naughtily GCC_CONFIG += --disable-lto-plugin in MCM. Just waiting on the zlib build then i'll trigger this one

    i should add that the MCM config for that arch is not x86_64
    Don
    @AwesomeIbex
    @thomcc So i've spent most of the morning checking linking issues and im certain i have zlib. Do you have any other ideas?
    Don
    @AwesomeIbex
    looks like its the same issue as this, rust-bio/rust-htslib#231
    2 replies
    so not a rusqlite issue i think
    Jose D Robles
    @josediegorobles
    hi everybody, i'm interested in have some database on wasm, it's possible to use rustqlite on wasm?
    Sergi Delgado Segura
    @sr-gi
    Hi guys, is there any way of importing extended error codes from FFI? I may be missing something but looks like they are not public (e.g. SQLITE_CONSTRAINT_FOREIGNKEY or SQLITE_CONSTRAINT_PRIMARYKEY)
    Thom Chiovoloni
    @thomcc
    if you use the bundled build, they're probably there. if not, possibly modern_sqlite would do it too
    1 reply
    hmm
    i've been meaning to make that all cleaner