by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Hugo Lopes Tavares
    @hltbra_twitter

    we built a custom queueing solution on top of Redis (https://medium.com/yipitdata-engineering/our-queueing-solution-617974eb049) and have been using ElastiCache for almost 7 years now. the cost and maintenance was a problem, so I decided to investigate and experiment with an EBS-based redis clone (https://medium.com/yipitdata-engineering/dredis-our-disk-based-redis-implementation-b525b1cfa74e).

    the limitations aren't great for our use case so we currently have a hybrid of redis/dredis in production

    I'm very interested in your implementation details, I want to see how you solved some of the challenges I faced. is there any plan for a blog post or making the code available online?
    Mattia Bianchi
    @mattiabi
    Hello Hugo, thanks. It's good to hear you are also worked on a Redis implementation.
    We will definitely read your blog posts and look at your code (maybe we can get some ideas :D ).
    Scripting support is not on the top of our feature list. But still we plan to implement it.
    We are trying to prioritize features according to community feedback. And your scripting feedback is noted.
    Hugo Lopes Tavares
    @hltbra_twitter
    if I'd redo the work I'd think much harder about using Python btw. there's no concurrency support besides using select() and it makes it annoying to add good concurrency without rewriting most of it. I followed the redis design of a single-thread and everything serial, it makes a wonderful and simple implementation, but the drawbacks were huge because of leveldb + EBS latency and key design, even without fsync()'ing
    Mattia Bianchi
    @mattiabi
    We plan to write more blog posts about technical details in the upcoming days.
    Yes, I'm not a python expert, but what I read and know from some colleagues it's difficult to scale a python process to support more concurrency.
    Hugo Lopes Tavares
    @hltbra_twitter

    @mattiabi: do you have benchmarks available?

    dredis has become a bottleneck when we have zsets with millions of items and try operations that end up being O(N) where N is a large number of keys in the storage layer, such as ZCOUNT -inf +inf

    Mattia Bianchi
    @mattiabi
    Of course we did many benchmarks while implementing the database but they are not ready to be published without a reproducible setup. You know, with a biased benchmark you can achieve what you want to see :)
    Hugo Lopes Tavares
    @hltbra_twitter
    cool, thanks. looking forward to more details. good luck with the product! thanks for the answers too
    Mattia Bianchi
    @mattiabi
    thanks. I hope we will post more in our blog...
    karan-c3
    @karan-c3
    hey I have just Deployed Lambda store on Netlify functions I get following errors
    failed - Client network socket disconnected before secure TLS connection was established
    most of the time
    Sven Anderson
    @svenanderson
    Hi @karan-c3 . Do you enable TLS on your database?
    I have just written an app on netlify: https://medium.com/lambda-store/who-said-that-a-tweet-guess-game-implemented-with-serverless-stack-on-netlify-7ae7e9cb0273
    I have not seen a problem, but I will try to reproduce your case, if you give more details
    karan-c3
    @karan-c3
    Hey I am using Redis client as singleton global object
    to reuse the connection
    Mattia Bianchi
    @mattiabi
    Karan, did you enable TLS on Lambda Store database?
    karan-c3
    @karan-c3
    yes
    Mattia Bianchi
    @mattiabi
    And also in Redis client configuration?
    karan-c3
    @karan-c3
    yes
    Mattia Bianchi
    @mattiabi
    which client library are you using?
    karan-c3
    @karan-c3
    tls: {}
    node-redis
    Sven Anderson
    @svenanderson
    so you do not close the connection, right?
    do you use context.callbackWaitsForEmptyEventLoop = false;
    karan-c3
    @karan-c3
    yes
    Sven Anderson
    @svenanderson
    as described here: https://dev.to/lambdastore/reuse-redis-connection-in-aws-lambda-5ena
    karan-c3
    @karan-c3
    no
    I have not used context.callbackWaitsForEmptyEventLoop = false;
    Sven Anderson
    @svenanderson
    can you check the blog: https://dev.to/lambdastore/reuse-redis-connection-in-aws-lambda-5ena
    karan-c3
    @karan-c3
    ya I've looked into that
    Sven Anderson
    @svenanderson
    the lambda/netlify function hangs until you close connection then it times out. this can be the problem
    Mattia Bianchi
    @mattiabi
    It might be better to close the conn when function quits and re-open next time if your app is not too much latency sensitive.
    Sven Anderson
    @svenanderson
    yes, closing the connection is the cleanest way to do it
    karan-c3
    @karan-c3
    ya I was thinking about that, It would require major refactoring
    ya
    but I'll do that
    Thanks
    Mattia Bianchi
    @mattiabi
    You can set context.callbackWaitsForEmptyEventLoop = false; to verify whether this was the issue...
    karan-c3
    @karan-c3
    I'll close the connection after every function call, latency is not an issue for us coz we are already saving lot of time by caching
    Thank you
    everyone
    I really like Lamda store coz we were creating JAM stack eCommerce on a tight budget. And Lambda store suits our stack due to its serverless nature. I am very happy with it
    Keep up the good work
    Mattia Bianchi
    @mattiabi
    good to hear that!