These are chat archives for ReactiveX/RxJava
RxJava – Reactive Extensions for the JVM – a library for composing asynchronous and event-based programs using observable sequences for the Java VM.
Hi, we are having problems getting our head around a backpressure problem. We want to pull files in parallel from a remote and slow location in order to analyze them in the next step with a computation bound analyze step, which should also be done in parallel. Unfortunately the analyzing step is slower than getting the files, which leads to our disk running out of memory. Therefore we need to adjust backpressure settings to slow down the producer of file downloads?
Our code looks something like this:
Flowable.fromIterable(fileListing) .flatMap(file -> Single.fromCallable(file -> getFileFromRemoteLocation(file)) .subscribeOn(Schedulers.io()) ) .map() // backpressure needed here? e.g. after 10 files stop getting more .flatMap(localFile -> Single.fromCallable(file -> analyzeFile(file)) .subscribeOn(Schedulers.computation()) )
How do we communicate appropriate backpressure in this case?
Flowable.fromIterable(files) .flatMap(file -> Flowable.fromCallable(() -> getFileFromRemoteLocation(file)) .subscribeOn(Schedulers.io()) .observeOn(Schedulers.computation()) .map(f -> analyzeFile(f)) , 10) ...