Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Leonhard Gruenschloss
    @lgruen
    @110y Will keep you posted! Related to wrench, did you see cloudspannerecosystem/wrench#21
    ?
    Yuki Ito
    @110y
    @lgruen Thank you for notifying. I've just merged now!
    Leonhard Gruenschloss
    @lgruen
    @110y Finally got all the approvals! Can we try the repo transfer again, please?
    Yuki Ito
    @110y
    image.png
    @lgruen Yes! I've already requested the transfer from 110y/yo to lgruen/yo. Could you reconfirm it?
    Leonhard Gruenschloss
    @lgruen
    @110y The previous request (from March 4) unfortunately has already expired ("Oops! That repository transfer has expired.")
    Yuki Ito
    @110y
    @lgruen Got it, I've just re-sent the invitation now!
    Leonhard Gruenschloss
    @lgruen
    @110y Awesome, thank you very much! I've added you as an admin. Can you please take a look to make sure everything looks alright? https://github.com/cloudspannerecosystem/yo
    I'll take a look at the CircleCI integration tomorrow if that's okay. Is there anything I should look out for? Is all that's needed this set of environment variables? (https://github.com/cloudspannerecosystem/yo/blob/master/.circleci/config.yml#L48)
    If you could send another description snippet for the website, that would be great!
    Yuki Ito
    @110y
    @lgruen Thanks a lot! Everything looks good!
    Is all that's needed this set of environment variables?
    Yes, will work fine.
    If you could send another description snippet for the website, that would be great!
    I will send PR soon!
    Leonhard Gruenschloss
    @lgruen
    @110y Thanks! I've configured the CircleCI environment variables, but it looks like yo expects an existing database? (https://circleci.com/gh/cloudspannerecosystem/yo/91)
    Masahiro Sano
    @kazegusuri
    yeah. Now we haven't automated the database creation. We use an existing database in a CI.
    Leonhard Gruenschloss
    @lgruen
    @kazegusuri Thanks! Could you send me instructions on how to generate that database in our Cloud Spanner instance, please?
    Yuki Ito
    @110y
    @lgruen I've send the PR which adds description about yo to ecosystem website. Please take a look when you have time! cloudspannerecosystem/cloudspannerecosystem.github.io#8
    And next, we would like to transfer spool: https://github.com/gcpug/spool which is a database pooling tool for CI. Could you please initiate the transfer process for it?
    Leonhard Gruenschloss
    @lgruen
    @110y Thanks very much! Added a minor comment. I'll take a look at spool tomorrow!
    Yuki Ito
    @110y
    @lgruen Thanks! Fixed it.
    Leonhard Gruenschloss
    @lgruen
    @110y Thanks very much! Merged.
    Vyacheslav Kim (Kane)
    @kanekv
    Hello everyone. What is a good place to ask generic spanner questions? Is there a spanner community?
    Leonhard Gruenschloss
    @lgruen
    @kanekv There are a few options, e.g. Stack Overflow -- see https://cloud.google.com/spanner/docs/getting-support
    Mayur Kale
    @mayurkale22
    Hi, could someone review and merge this? cloudspannerecosystem/website#5. It is very simple.
    Leonhard Gruenschloss
    @lgruen
    @mayurkale22 No problem, done!
    Mayur Kale
    @mayurkale22
    @lgruen thank you :)
    flowerinthenight
    @flowerinthenight
    Hi, I have a repo https://github.com/flowerinthenight/spindle. It's a public version of what we use in production at the moment. I wonder if this repo can join the ecosystem?
    skuruppu
    @skuruppu
    Thanks @flowerinthenight for reaching out. Just wanted to let you know that we won't be able to check it out till early Jan as most of the team is away for the end of year holidays. Hope that's ok.
    flowerinthenight
    @flowerinthenight
    No worries. Thanks.
    Vyacheslav Kim (Kane)
    @kanekv
    Hello, is it a good place to talk about generic spanner stuff?
    skuruppu
    @skuruppu
    @kanekv you certainly can discuss general spanner topics here. To get a broader audience, especially if you have questions, you may still want to consider having a discussion in place like StackOverflow.
    skuruppu
    @skuruppu
    @flowerinthenight, really sorry for the delay. The repo you linked to is very interesting. Just wanted to ask you a couple of questions to help us decide.
    1. Would you be able to give some background on why you're using this approach in your production app? Better still, if you could document some use cases in the README, that would be great.
    2. Why did you choose Spanner for this over other databases?
    flowerinthenight
    @flowerinthenight

    @skuruppu No problem at all.

    1. Most of our teams employ some sort of distributed processing in our services, and distributed locking is quite a common technique in this area. We use some of the more well known ones such as Zookeeper and Consul. These systems are quite difficult to maintain by themselves though, so any option that are easier when it comes to maintenance are welcome to us. Redis is also common in our backends and I even have a Redis-based distributed locker lib https://github.com/flowerinthenight/kettle that we use as well. But the Redis version is not really that reliable as described here. This would lead me to:

    2. We use Spanner heavily as one of our main databases, among others. So if I could do distributed locking using Spanner itself, I don't have to maintain another Zookeeper just for locking. I think TrueTime is crucial here and may even be more reliable than the Redis version, time will tell. So far, our Zookeepers are the most reliable but I think that this library can be as reliable as well, which it is, so far.

    skuruppu
    @skuruppu
    Thanks heaps for the details @flowerinthenight. I will take a closer look at the repo next week and also figure out the steps needed to do an import. Hope it's ok if I get back to you late next week.
    skuruppu
    @skuruppu
    @flowerinthenight I took a look at your repo and it'll be a nice tool to be available through the ecosystem. Before we proceed, there are a couple of things:
    1. I still need to figure out the official process we should use to move the repo. I'll get back to you with details next week.
    2. We'll need to add some tests. I'm not sure if it's important to run the tests against the production backend or whether it's possible to run it against the Cloud Spanner Emulator. Is this something you've looked into?
    3. We'll need to do a code review. We can do this before or after the move. @olavloite would you be able to take a quick look at the code and provide any feedback you have about the approach?
    Knut Olav Løite
    @olavloite
    @flowerinthenight Thanks for the contribution. I really like the idea of this. I've been going through the code (not in all detail yet), and I had a couple of questions:
    Knut Olav Løite
    @olavloite
    1. (More of a general question) If I understand it correctly, the current implementation allows multiple records for the same lock to exist in the table, and uses a process of adding new records when a new lock is acquired and deletes old records that are no longer valid. Would it also be possible to change this to an implementation that only allows one record per lock (enforced with a unique index or even through the primary key of the lock table)? I think that would make the implementation slightly simpler and also more robust, as Cloud Spanner could then be used to not only supply the lock timestamp but also the uniqueness of the lock. It would also make it possible to use mutations instead of DML, which again allows for the use of InsertOrUpdate mutations. That could simplify the current implementation that currently has specific handling for the first insertion, and the update method that executes separate UPDATE and DELETE statements. Or am I missing something here that makes such an implementation impossible / unfeasible?
    This message was deleted
    1. Could you look into breaking up the current Run function into a couple of separate functions? The current nesting of a combination of functions and loops is quite deep and that makes it difficult to grasp exactly what is going on there.
    flowerinthenight
    @flowerinthenight

    @skuruppu For #2, yes I was looking into the possible use of the emulator for tests but haven't got the time to try and implement it yet.

    @olavloite Thanks very much. One record equals a single lock. Since the name column is the primary key, it should be unique. Throughout the duration of the lock lease, that single record (the one that's grabbed during initial) will be used, with the heartbeat and timestamp columns being updated periodically. During the initial iteration, it is important to use INSERT, and not InsertOrUpdate to make sure that only one client/process will succeed and all others fail. The same story with the succeeding attempts to grab the lock. It needs to use INSERT, not InsertOrUpdate. The cleanup part is actually not necessary. It's just there to keep the table contents small and will be done by the current lock owner. If removed, and, say, we have a lock duration of 1sec, a new record will be added to the table every second. This bit probably needs a better implementation.

    For the 2nd one, yes you're right. It could do with a bit of breaking down. I'll make some improvements hopefully sooner when I can find some spare time. I just have to make sure that changes will not affect our current production.

    skuruppu
    @skuruppu

    Thanks @flowerinthenight, we would like to have some tests before we publish it in the repo.

    For the publishing process, I think we can create a repo in the ecosystem then do a repo move. I will have to go through some internal processes to create the repo. Note that you and anyone else that has committed to the repo will need to sign a Google CLA (https://cla.developers.google.com/about/google-individual). Would this be ok with you and the other contributors?