it's absolutely necessary for healthy thread pool usage to use a blocking EC correctly; the recommended cats-effect setup (and the way that the ecosystem works) is that you use a fixed-size (= cores) thread pool as your primary EC that backs your
ContextShift, and a cached thread pool for jvm-thread-blocking calls, using evalOn to make sure those calls happen on the thread blocking pool
if you do thread-blocking calls without using
evalOn (or related) to put it on the right pool, you risk severely hampering the application's efficiency
If you would like more maintainers, @gafiatulin and I volunteer
Hi @jgogstad sorry for the delay, its been hard to sync up since Matt is currently the only one with merge access and access to LendUp's GPG keys... I'll try to sync up with him and get this merged and published ASAP. We will also be moving this to a separate GitHub org so we can open it to more maintainers. Once we merge and publish these last PRs we will go ahead and do that.
ah. looks like an easy fix
Sorry again for the delay. Ditto what Rolando said about moving this to another org, we need more maintainers besides me :sweat:
@gafiatulin would you mind pulling master into your branch? lendup/fs2-blobstore#49
Looks like Travis is using the travis.yml from your branch, not master. I thought it would merge master first and use travis.yml from there. Just trying to get a green build before merging
@jgogstad can you fix conflicts on lendup/fs2-blobstore#50 ? I will merge this today. Trying to get this one in and #52 before doing a release
also @gafiatulin for lendup/fs2-blobstore#52
they might be straightforward but you probably have more context
they might be straightforward but you probably have more context
you’re not apart of the fs2-blobstore org, @matthewgraf, added you as collaborator, but I still can’t add you as a reviewer for PRs
can you have a look at fs2-blobstore/fs2-blobstore#2 ?
Hi. I’ve just put up a PR to update fs2 to 2.2.2. It means dropping scala 2.11 support but brings the fs2 version in line with http4s 0.21.x. lendup/fs2-blobstore#60
@DougC take a look at the repo in the new org fs2-blobstore... https://github.com/fs2-blobstore/fs2-blobstore
@matthewgraf can you add a big not on LendUp repo saying that we moved?
Btw has anyone seen this problem when using bufferedPut on an S3Store ?Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
I'm wondering if this is a problem specific to our usage of S3Store or an underlying problem with S3Store, involving puts on large files
I saw this brought up on Stackoverflow and elsewhere, I'll dig into it but was just curious to see if anyone else encountered this
Maybe I'm misunderstanding
bufferedPut. Does it write the entire file to the local filesystem before writing to the output Path?
Also worth noting this flow has been untouched and working for a while - it appears once we are dealing with larger files (e.g. 5 GB in this case) it becomes a problem
wonder if there is a workaround.. would be nice to stream data into s3 w/o having to write a full file to some other store
Is this still the right channel for discussion? It’s still associated with the lendup repo. I’m doing some work with the sftp store and just hit a problem listing files that I think is resolved by this fs2-blobstore/fs2-blobstore@23375b1
Would it be possible to push out a release with that change?
@DeanZi : you configure this on the
TransferManager, see the main readme in the repo. E.g. val awsCredentials = new BasicAWSCredentials(accessKey, secretKey)
val credentialsProvider = new AWSStaticCredentialsProvider(awsCredentials)
val transferManager = TransferManagerBuilder.standard().withS3Client(AmazonS3ClientBuilder.standard.withCredentials(credentialsProvider).withRegion(region).build) .build
I am right in thinking this comment in the
Store[F].put scaladoc is out of date? It looks like S3Store has support for multipart uploads when the size of the upload is unknown.
@bpholt yes, this comment is out of date since fs2-blobstore/fs2-blobstore#40 got merged. Let me update that.
