It's mostly batch processing and for each job I allocate a lazy connection. Then I use queryStream on x amount of rows to start processing the first row as soon as it's there. https://github.com/friends-of-reactphp/mysql#querystream
This is exactly how I was going to manage huge load of queries (increase amount of connections and split queries between them to process concurrently).
queryStream in combination with reactphp-flux is amazing. https://github.com/clue/reactphp-flux
I haven't tried it yet, but now I would like to. Previously I created similar low-level functions (with content lockers and concurrency limiters) for batch operations by myself.
@iorsa what is your use case? MySQL indexes are good (enough) if applied properly.
I guess so. I don't have great knowledge and experience (yet) about mysql optimizations. I've recently started to practice with optimizations for big tables and indexes were the first thing that helped a lot. Though, for some queries the processing time is still high (from 0.7 sec and up to 2.5 sec and more) even with indexes (most of results for same query structure have <0.3 seconds processing time, depending on search criteria).
I have a table with 67 columns where ~18 of them are indexed in one column and combined multi-column indexes (~10, from 2 and up to 14 columns). There are ~2 millions of rows that in total bring 7.6 GB of data. All indexes size ~3 GB. I've made a research and tests that showed that some queries with the use of combined multi-column indexes are faster than with merged index that consist of the same indexes in the same order, so I've created various combined multi-column indexes for such most popular queries. I also use merging single-column indexes for queries that showed faster results without use of combined index.
Mostly there are ~5 columns used in query WHERE clause and their processing time ~0.1-2.5 sec (more often <1 sec).
I've also noticed that subquery with INNER JOIN made query faster up to 50 times than only with the us of indexes, so I've also started restructuring queries to use subqueries where it brings speed (it is also reduces CPU usage).
There are also few virtual columns ( STORED) based on simple arithmetic operations and 1 with hash on 2 columns that also showed faster query processing time.
querystream will help for that. The problem is with slow processing for queries with multiple search criteria. I only use subquery for search inside same table and it really helps much better than indexes. Looks like I need to learn how to do better indexing (and limit indexes as well) or maybe increase RAM and store indexes there? May we a little continue in PM?
$file_stream = new Clue\React\Csv\AssocDecoder(
new React\Stream\ReadableResourceStream(
fopen('data.csv', 'r'),
$loop
)
); to queryStream that will pass the result to Clue\React\Flux\Transformer, that will process that data from queryStream further?
I've got an error:
Fatal error: Uncaught TypeError: Argument 1 passed to Clue\React\Csv\AssocDecoder::pipe() must implement interface React\Stream\WritableStreamInterface, instance of React\Promise\Stream\UnwrapReadableStream given
Is there some higher level event manager/dispatcher like the one from symfony for ReactPHP or can I use just that one?
Anyone a good suggestion? I've to fire an event like SomethingCreatedEvent after a Http call and react on it (which may be a another long running function) but if I use the symfony implementation it won't be really async I suppose or is this fast enough to combine with reactphp
3 replies