Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Noah Watkins
    @noahdesu
    src/include/zlog/backend.h
    so here is the current list of feedback on the PR
    Noah Watkins
    @noahdesu
    1. report errors for invalid CLI invocations (e.g. like leaving off log name --> segfault)
    2. have dump report trim/filled position states
    3. clean up the commit log a bit after all the dev work is done. for example, it's confusing to see all the intermediate development steps (wip, then remove rename, etc...) or add testing then fix testing. i'd squash the commits down into logical commits. unfortuantlely there isn't any right way to do this, it's just a look and feel kind of thing.
    other than those things, the PR looks really good
    Victor Sui
    @vsui
    how would I get -ERANGE to show up for read?
    Noah Watkins
    @noahdesu
    what do you mean?
    like to test it?
    Victor Sui
    @vsui
    also when I fill/trim past the end and dump I get ENOENTs starting at the end of the written log
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log create newlog
    zlog.head.2aa01f68-8fdf-4c38-bbc0-0c9c33a617d3
    zlog.data.2aa01f68-8fdf-4c38-bbc0-0c9c33a617d3
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log append newlog a
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log append newlog b
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log append newlog c
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log trim newlog 1
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log trim newlog 5
    vsui zlog (cli) $ ./bin/zlog --backend lmdb --db-path ../lmdb/ log dump newlog
    0: 61
    1: invalidated
    2: 63
    log::Read -2
    yeah to test it
    Noah Watkins
    @noahdesu
    ahh
    what happens if you try to read 10000
    or whatever
    like read 20
    Victor Sui
    @vsui
    ENOENT again
    should I add a read command to just print a specific entry
    Noah Watkins
    @noahdesu
    ahh i see... for -ENOENT dump out "pos: object not found" -- this should be -ERANGE, but it's getting reported incorrectly inside zlog. I can fix that
    should I add a read command to just print a specific entry yeh that'd be awesome
    you could even report "pos: object not found (see cruzdb/zlog#316)" haha
    Victor Sui
    @vsui
    lol
    Victor Sui
    @vsui
    Can you review the last few commits and then I can start fixing up the commit history
    Noah Watkins
    @noahdesu
    @vsui it looks good
    Noah Watkins
    @noahdesu
    merged. thanks a lot!
    i think it is really cool that you reached out in twitter DM about OSS and then go involved and put together a non-trivial patch. definitely feel free to say no, but if you don't mind, I'd like to tweet out a little screenshot of DM and the subsequent PR to show #opensource how easy it is to get involved with some projects.
    Victor Sui
    @vsui
    yeah thats fine
    Noah Watkins
    @noahdesu
    cool :)
    Victor Sui
    @vsui
    Anything else you think I could work on?
    Noah Watkins
    @noahdesu
    Yeh definitely. Any area you are interested in?
    Victor Sui
    @vsui
    #237 and #305 sound interesting to me
    What do you think?
    Noah Watkins
    @noahdesu
    #237 could be fun to work on but it’s really low priority, and #305 is ceph specific—it’s actually not really an action item so much it is just need to meet w some ceph devs to ask some questions.
    I think that implanting delete / gc would be helpful and would also require more design work to make sure it’s correct
    Do you have any experience with TLA+?
    Lemme look at my notes. I think there are some other things that I’m not thinking of right now
    Victor Sui
    @vsui
    I’ve listened to a podcast about it lol
    Would be down try to learn TLA+ if there’s anything that needs to be done with it, it sounded really cool
    Noah Watkins
    @noahdesu
    yeh that's about my level of knowledge too. i've hacked with it a bit, but i'm not very good. one goal i've had is to formally model zlog to show that it is correct. this is useful to build confidence in the implementation, but also to test out new optimizations in the model before committing to building them in C++. so that's definitely a project that is cool (plus formal modeling a hot topic right now).
    so i think (1) tla+ modeling (2) or something like: log deletion, garbage collection, monitoring tools, integrations... generally working on something to advance zlog to its 1.0 goal or (3) cruzdb type stuff. i can expand on (2) tomorrow--i just need to go look at my notes
    Victor Sui
    @vsui
    Ok yeah those all sound cool. I’ll look into TLA+ and lmk about (2) because those sound cool too
    Noah Watkins
    @noahdesu
    i'm working through it
    you can also consider log delete / gc to be absolutely necessary things that need to be done, but they will require some design work--not as straightforward as building out a CLI.
    but that might be a benefit :)
    Victor Sui
    @vsui
    Yeah I think those are probably good for me to focus on for now and I can look at the TLA+ in the background
    Noah Watkins
    @noahdesu
    sure sounds good. i have some thoughts about delete / gc, so when you have some time we can talk about it.
    Victor Sui
    @vsui
    Ok I can take any tips whenever, just lmk
    Noah Watkins
    @noahdesu
    the key thing with delete is that i think we want to separate it into two actions. (1) delete: which is a logical operation and (2) garbage collection where we actually remove the log storage. For (1) i think we want to (a) mark in the metadata of the head object that the object is deleted and then (b) delete the link object. If we add a creation timestamp or "last opened" timestamp to the head object, then for (2) we can add a mode to the CLI to look for deleted objects older than N days and actually perform the deletion.
    Victor Sui
    @vsui
    hey noah, holidays have been busy + I've moved recently but just wanted to let you know that I intend to get back to this in the coming weeks :)
    Noah Watkins
    @noahdesu
    ok sounds great! looking forward to working with you again soon
    ddorian
    @ddorian
    are you using something like this for panda, or is it strict local-store like scylladb etc