Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 10:56
    Favy-source commented #431
  • 09:37
    clue labeled #256
  • 07:58
    mr-older closed #256
  • 07:57
    mr-older commented #256
  • Jun 20 21:17
    WyriHaximus commented #408
  • Jun 19 11:41
    clue commented #256
  • Jun 19 11:06
    WyriHaximus commented #89
  • Jun 19 10:16
    mr-older commented #256
  • Jun 19 06:48
    mr-older opened #256
  • Jun 17 08:22
    mityukov commented #408
  • Jun 17 07:22
    d0niek commented #408
  • Jun 17 07:21
    d0niek commented #408
  • Jun 16 11:40
    mityukov opened #408
  • Jun 15 13:56
    clue commented #216
  • Jun 09 19:00
    Nedum84 commented #444
  • Jun 09 05:43
    clue commented #444
  • Jun 07 05:38
    stereomon edited #158
  • Jun 06 17:38
    WyriHaximus commented #89
  • Jun 06 17:07
    WyriHaximus closed #77
  • Jun 06 17:07
    WyriHaximus commented #77
Marc Morera
@mmoreram
Hello people! Do you have any clue about how the reactphp browser can reuse TLS connections across requests? I've been experimenting some problems with elasticsearch requests, and what I found is that they are caused by DNS resolving and TLS negotiation
I'm trying to solve the DNS part by resolving it as soon as the application starts (I'm not sure if the browser automatically cache it)
Marc Morera
@mmoreram
(reuse SSL connections)*
Christian Lück
@clue
@mmoreram I'm not sure I understand what problem you're seeing, can you provide more information?
The HTTP client implementation does not currently leverage HTTP keep-alive connections (see issue tracker), but we're looking into this for an upcoming version 👍
Marc Morera
@mmoreram
@clue sure!
Christian Lück
@clue
Either way I don't see how/why this would cause any issues, as it's an entirely optional feature
Marc Morera
@mmoreram
We have a server A and a server B. Server A should request server B as fast as possible (2ms i better than 10ms). Server A requests B by using TCP requests, and is always the same DNS resolution
Each time A requests B, a new socket is created, so if we're using HTTPs, that means a full TLS handshake resolution each time, what means a lot of "lost" time
(I'm sorry @clue, sometimes I feel a bit lost when I talk about these topics. I'm not an expert at all)
By reusing the same connection once and again, or a pool of connection, server A would avoid reconnecting once and again, increasing so much the performance in terms of Server A response time
In my case, having an internal latency of 20ms is an issue (at least I consider this an issue in my specific context)
Christian Lück
@clue
@mmoreram No worries
DNS records specify a TTL (commonly 3600 seconds), these records will be cached in memory on the client side by default, so it would only send this DNS query once for multiple connections to the same hostname.
HTTP keep-alive is definitely on my wishlist and would help prevent this unneeded TCP/IP handshake and TLS negotation 👍
Definitely a nice improvement in the future 👍
Marc Morera
@mmoreram
that's it. I think that we talk about the same then :)
Thanks for your response. I appreciate it so much
Christian Lück
@clue
@mmoreram Happy to help! 🤝
Ramon Ennik
@cyrnetix
Is there some higher level event manager/dispatcher like the one from symfony for ReactPHP or can I use just that one?
iorsa
@iorsa
Hello. I'm curious if anyone is using reactphp and database queries pool and various other optimizations for managing queries for tables with large amount of columns and rows. When I'm doing SELECT even with indexing it is still relevantly slow. I am not an expert in databases, but it looks like memory cache for popular queries and queries pool can help to speed it up. @geertvanbommel If I remember it correct, you have some great experience with reactphp and database mass I/O.
iorsa
@iorsa
Besides queries pool it seems like having connections pool is also a nice idea.
Geert Van Bommel
@geertvanbommel
@iorsa connections pool is being worked on. So far my use case has been different
It's mostly batch processing and for each job I allocate a lazy connection. Then I use queryStream on x amount of rows to start processing the first row as soon as it's there. https://github.com/friends-of-reactphp/mysql#querystream
queryStream in combination with reactphp-flux is amazing. https://github.com/clue/reactphp-flux
Geert Van Bommel
@geertvanbommel
@iorsa what is your use case? MySQL indexes are good (enough) if applied properly.
iorsa
@iorsa
@geertvanbommel Thanks for reply!

It's mostly batch processing and for each job I allocate a lazy connection. Then I use queryStream on x amount of rows to start processing the first row as soon as it's there. https://github.com/friends-of-reactphp/mysql#querystream

This is exactly how I was going to manage huge load of queries (increase amount of connections and split queries between them to process concurrently).

queryStream in combination with reactphp-flux is amazing. https://github.com/clue/reactphp-flux

I haven't tried it yet, but now I would like to. Previously I created similar low-level functions (with content lockers and concurrency limiters) for batch operations by myself.

@iorsa what is your use case? MySQL indexes are good (enough) if applied properly.

I guess so. I don't have great knowledge and experience (yet) about mysql optimizations. I've recently started to practice with optimizations for big tables and indexes were the first thing that helped a lot. Though, for some queries the processing time is still high (from 0.7 sec and up to 2.5 sec and more) even with indexes (most of results for same query structure have <0.3 seconds processing time, depending on search criteria).
I have a table with 67 columns where ~18 of them are indexed in one column and combined multi-column indexes (~10, from 2 and up to 14 columns). There are ~2 millions of rows that in total bring 7.6 GB of data. All indexes size ~3 GB. I've made a research and tests that showed that some queries with the use of combined multi-column indexes are faster than with merged index that consist of the same indexes in the same order, so I've created various combined multi-column indexes for such most popular queries. I also use merging single-column indexes for queries that showed faster results without use of combined index.
Mostly there are ~5 columns used in query WHERE clause and their processing time ~0.1-2.5 sec (more often <1 sec).
I've also noticed that subquery with INNER JOIN made query faster up to 50 times than only with the us of indexes, so I've also started restructuring queries to use subqueries where it brings speed (it is also reduces CPU usage).
There are also few virtual columns ( STORED) based on simple arithmetic operations and 1 with hash on 2 columns that also showed faster query processing time.

Geert Van Bommel
@geertvanbommel
@iorsa It sounds like you already have way too many indexes. 2 million rows is not a lot. Depending on your hardware and MySQL config your indexes might not fit in memory making it equally hard for MySQL to use them. Total query time also depends on how many rows you return of these 2 million. The EXPLAIN method is your friend to see how much of the data is estimated to get scanned. The fact that a JOIN helped, indicates MySQL was able to use the primary key or index from that other table. And yes, subqueries are not bad if used correctly. Another common mistake is joining too many tables inside a subquery and do a group by on the subquery where the tables could easily have been joined after the group by. Rule 1: KISS. 2 separate queries can be better than 1 joined. Rule 2: rewrite queries and use EXPLAIN to get the best performance. Rule 3: only add indexes if no option is left.
Let's not hijack this thread more about MySQL stuff, although I could talk for hours about it :D
iorsa
@iorsa
@geertvanbommel Agree, this is not the right place for mysql discussions, except if it is related to reactphp like I originally interested. I'll try to use https://github.com/clue/reactphp-flux with https://github.com/friends-of-reactphp/mysql#querystream as you suggested. I'm also looking for the way how to use caching with reactphp and mysql popular queries. My basic server params (4 CPU, 8GB RAM, Debian 10, 10.3.27-MariaDB-0+deb10u1-log). I tried to use mysql/mariadb query cache, but with it disabled the results were better. Sometimes I need to return hundred thousands of rows and I believe querystream will help for that. The problem is with slow processing for queries with multiple search criteria. I only use subquery for search inside same table and it really helps much better than indexes. Looks like I need to learn how to do better indexing (and limit indexes as well) or maybe increase RAM and store indexes there? May we a little continue in PM?
iorsa
@iorsa
My bad.. InnoDB buffer pool already stores indexes in RAM :D
iorsa
@iorsa
May I ask how is it possible to pass data from $file_stream = new Clue\React\Csv\AssocDecoder( new React\Stream\ReadableResourceStream( fopen('data.csv', 'r'), $loop ) ); to queryStream that will pass the result to Clue\React\Flux\Transformer, that will process that data from queryStream further?

I've got an error:

Fatal error: Uncaught TypeError: Argument 1 passed to Clue\React\Csv\AssocDecoder::pipe() must implement interface React\Stream\WritableStreamInterface, instance of React\Promise\Stream\UnwrapReadableStream given

Any example, please? I want to use data from csv file to create mysql query and update database table
iorsa
@iorsa
I'm just doing $file_stream->pipe($stream)->pipe($transformer);, where $stream = $db_connection->queryStream($query);
Christian Lück
@clue
@iorsa The queryStream() method returns a readable stream that emits row data from your query
You're trying to pipe a readable stream into a readable stream, hence the error (you should pipe a readable stream into a writable stream)
What exactly are you trying to achieve?
iorsa
@iorsa
@clue I want to use stream for reading data from csv file and pass every line from csv to other stream that will make sql query based on csv line data
And after first sql (select) query I would like to make one more query to update the table data
Christian Lück
@clue
You can use https://github.com/clue/reactphp-flux#streaming if you want to run one query per row from your CSV data, but note that depending on your data structure this may be somewhat inefficient
Other than that: KISS. Start with consuming data from your CSV stream and sending it off to your database. Once this is sorted, you can always pipe this into another writable destination stream for your update statements
Batching multiple rows (let's say each 100 rows) and then combining this with data from your database would likely be significantly faster, but does incur some additional implementation complexity
If there's anything we can help with, reach out ✌️
Ramon Ennik
@cyrnetix

Is there some higher level event manager/dispatcher like the one from symfony for ReactPHP or can I use just that one?

Anyone a good suggestion? I've to fire an event like SomethingCreatedEvent after a Http call and react on it (which may be a another long running function) but if I use the symfony implementation it won't be really async I suppose or is this fast enough to combine with reactphp

3 replies
iorsa
@iorsa
@clue Thanks! After few attempts I've finally learned how to work with that various streams and why there were errors. Actually piping is very simple and smooth process, but takes time to understand the details.