Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 08 08:51

    ioquatix on v1.24.0

    (compare)

  • Dec 08 08:49

    ioquatix on v1.17.1

    (compare)

  • Dec 08 08:27

    ioquatix on v1.24.0

    (compare)

  • Dec 08 07:37

    ioquatix on master

    Update links to travis. (compare)

  • Dec 08 07:12

    ioquatix on master

    Update documentation. (compare)

  • Dec 08 06:55

    ioquatix on master

    Rename `*args` -> `*arguments`. Improve spec determanism. (compare)

  • Dec 08 06:50

    ioquatix on master

    Improve spec determanism. (compare)

  • Dec 08 06:35

    ioquatix on master

    Explicit support for embedding … Remove `Reactor#timeout`. Renamge `*args` -> `*arguments`. (compare)

  • Dec 07 14:06

    ioquatix on master

    Add `Reactor#pause` and `Task#p… (compare)

  • Dec 07 01:00

    ioquatix on master

    Drop support for Ruby 2.3. (compare)

  • Dec 05 16:01

    ioquatix on master

    Improved syntax/comment in spec… (compare)

  • Dec 05 15:52

    ioquatix on master

    Improve implementation of queue… (compare)

  • Dec 05 15:17
    ioquatix closed #51
  • Dec 05 15:16
    ioquatix commented #51
  • Nov 08 16:00
    coveralls commented #52
  • Nov 08 16:00
    coveralls commented #52
  • Nov 08 15:33
    BMorearty synchronize #52
  • Nov 05 03:59
    ioquatix commented #52
  • Nov 05 03:58
    ioquatix commented #52
  • Nov 05 03:42
    BMorearty commented #52
Phil Pirozhkov
@pirj
@ioquatix Hey, sorry for the late response, I check Gitter quite infrequently. Drop me a message on hello@fili.pp.ru
Bryan Powell
@bryanp
@ioquatix I saw the experiment with async-sequel and I'm wondering what your thoughts are in supporting it in a non-experimental sense. I might be interested in helping make it happen
Samuel Williams
@ioquatix
@bryanp there are some different angles to this discussion.
Every time I make these experimental wrappers, I run into issues where the complexity of the code makes the implementation pretty sub-standard, at least to my standards.
There are often threading issues (ActiveRecord) and/or design trade-offs which make it a bit messy (e.g. assumptions about concurrency and/or pooling).
I don't know a lot about dry-rb
but I feel like this needs to be 3 separate things:
low level asynchronous adapters for accessing databases that provides a consistent medium - high level adaptor, each database in a separate gem
(datamapper did something like this)
Then, above that, some kind of arel style abstraction which makes dealing with SQL a bit safer and more pleasent
then on top of that something like AR whichi helps users deal with models
So, something like
protocol-mysql, protocol-postgres, then async-database, then if required, maybe something like async-database-model
or maybe that can be abstract enough
I'm not quite sure I have time to implement all that unless companies are going to sponsor it
but to me, the reason why AR and Sequel are so hard to work with is actually because they combine so many layers and the code base doesn't really have a great separation of concerns
that makes it really hard to retrofit async into it
even stuff that should be relatively simple like asynchronous cursor based iteration... can be stupid tricky.
Samuel Williams
@ioquatix
my reason for writing these experimental wrappers is to understand the domain model a bit better... maybe something like dry-rb has the higher level components and if we just build the lower level adapters... then that could work?
I don't know enough about it.
Bryan Powell
@bryanp

Yeah, I hate to reinvent the wheel because so much work has gone into things like Sequel and they're fantastic libraries, but I see what you're saying. I implemented pakyow/data around Sequel, similar to rom but with some different design decisions. Let me go off on a tangent and I'll try to bring it back to the original discussion.

So, pakyow/data exposes a concept called data sources. A data source is just a wrapper around a value. For example there's a relational data source for dealing with relational data in an application. Let's focus on this use case for the purpose of this conversation.

Relational data sources define attributes, each consisting of a name and type, as well as named actions and queries. Predictably, state is mutated through actions and fetched through queries. To complete the mental model, queries just return raw values that can be mapped into other value objects.

Multiple adapters can be defined for each type of data source. For example, a sql adapter exists for relational data sources that is essentially a wrapper around Sequel datasets. When you write an action or a query you're really just interacting with Sequel datasets along with a few helper methods.

With all of that context out of the way, my point is that there are some rough edges using Sequel as it was never really intended to be used in this way. Some of that unfortunately leaks out to end users. It has me wondering if there's room for a library that doesn't try to be an end to end solution but instead something that a library like rom or pakyow/data could easily build on top of. I find myself in a similar situation to you trying to make these libraries async, but in a different sense.

Anyway this seems to align with what you laid out above. Use async-database-model for an out of the box library that an end user might want, or libraries could build their own abstractions on top of a slightly lower level async-database. Am I understanding your thinking here?

Samuel Williams
@ioquatix
Yeah
Thats pretty much it
So, let me explain some of the different use cases
You've kind of identified one use case, which is the high level business logic MVC type pileup
But there is at least one other use case...
big data
When you have a shit ton of rows, you actually need efficiency
instantiating a business record per row is rediculous
So, there is definitely a need for something that supports different layers
from the high level business logic to the efficient read rows, do this thing and write the result here x 300 million rows
You can see elements of this with ActiveRecord... when they have things like destroy_all vs delete_all - it's a bit of a mash up of different levels of abstraction
AREL is pretty amazing though, maybe we don't need to reinvent that?
Also, to be frank, I don't feel the need to implement 10 different database adapters
Samuel Williams
@ioquatix

# gems:
# `async-database-postgres`
# `async-database-mysql`

# `async-database` - abstract glue:

endpoint = Async::Database::Postgres.endpoint(username: ..., password: ...)

client = Async::Database::Client.new(endpoint)

client.execute("sql")

client.transaction do |connection|
end

connection = client.connect
connection.close
maybe a protocol-sql gem to implement the shared abstractions, perhaps depending on arel.
Bryan Powell
@bryanp
AREL is great, I agree. Sequel's approach is also very good, but AREL is great and there's probably more consensus around it. But... AREL is no longer a separate gem. It's unfortunately been absorbed into AR: https://github.com/rails/arel

Also, to be frank, I don't feel the need to implement 10 different database adapters

What do you mean by this?

Bryan Powell
@bryanp
Apologies for starting another thread, but what do you think about packaging the async redis adapter you PRd (redis/redis-rb#832) into a async-redis-driver gem? I know that async-redis exists, but using the "standard" redis-rb in an async way seems more appealing. And the maintainers are very clearly against new drivers as part of the core gem
Bryan Powell
@bryanp
@ioquatix Wanted to follow up on :point_up:... curious what your position is on building wrappers around existing gems (e.g. redis-rb, pg, mysql2) vs building brand new gems
Samuel Williams
@ioquatix
If these gems make assumptions about concurrency, it is very hard to use them in practice, even if they can work in theory. For example I wrote a redis-rb driver for async, but I'm not confident it won't deadlock. The concurrency model is interwoven with the interface. After a few years of mucking around with this stuff, splitting it by protocol-redis (pure protocol/interface) and async-redis (concurrency model) and then on top of async-container (parallelism model) is the best design possible.
Unfortunately the vast majority of code blurs the lines between these layers, making it almost impossible to reuse. I got pretty far with async-sequel though because it mostly works at the protocol layer with a model layer built on top. For the most part, you can wrap it in processes, threads or fibers (needs custom pool implememtation) and it works.
That being said, it's kind of messy. There are layering violations. There are issues in the underlying driver gems (e.g. mysql2 doesn't expose the C functions required for fully asynchronous behaviour).
I'm not saying that reinventing the wheel is desirable, but I am saying that sometimes it's the easiest option given how much chance there is of something going wrong or being difficult to work around.
I wish there is some kind of ORM layer which doesn't make any assumptions about concurrency. That would be refreshing.
Maybe dry-rb has some kind of design like this... but I didn't take a lot of time to investigate it yet. It seems like a pretty complicated set of tools.
Bryan Powell
@bryanp
Thanks for sharing your thoughts here. Too bad wrappers aren't possible in a clean way
What dry-rb libraries are you talking about? I'm not aware of anything database-related
Samuel Williams
@ioquatix
Do you know rom-rb? That’s what I was thinking of
Bryan Powell
@bryanp
Yes, that one. Great library, but still adds an orm (or rather an anti-orm) on top of sequel datasets