Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 11 2020 21:47
    TheJoshyGee removed as member
  • Mar 28 2020 10:50
    jgogstad closed #57
  • Mar 09 2020 14:32
    rolandomanrique commented #61
  • Mar 09 2020 10:07
    jgogstad commented #61
  • Mar 07 2020 11:34
    jgogstad opened #61
  • Mar 07 2020 11:18
    jgogstad commented #57
  • Feb 24 2020 11:45
    DougC closed #60
  • Feb 24 2020 11:45
    DougC commented #60
  • Feb 22 2020 12:53
    DougC commented #60
  • Feb 22 2020 12:42
    DougC opened #60
  • Feb 17 2020 09:01
    gafiatulin closed #59
  • Feb 17 2020 09:01
    gafiatulin commented #59
  • Feb 17 2020 08:59
    gafiatulin commented #58
  • Feb 17 2020 08:34
    gafiatulin closed #58
  • Feb 17 2020 08:34
    gafiatulin synchronize #58
  • Feb 13 2020 14:39
    gafiatulin edited #59
  • Feb 13 2020 14:39
    gafiatulin synchronize #59
  • Feb 13 2020 11:49
    gafiatulin synchronize #59
  • Feb 10 2020 20:30
    jakeyr removed as member
  • Feb 04 2020 09:49
    gafiatulin synchronize #59
Jostein Gogstad
@jgogstad
that’s correct. It’s a bit surprising as a client when I pass in an ExecutionContext that is bound to a variable named blockingExecutionContext and then blocking IO is done on whatever context the current thread is executing on
Gavin Bisesi
@Daenyth
I recommend updating to cats-effect 1.4 at least (if you're not on 2.x on master already) and using the Blocker interface, it adds a lot of clarity
it's absolutely necessary for healthy thread pool usage to use a blocking EC correctly; the recommended cats-effect setup (and the way that the ecosystem works) is that you use a fixed-size (= cores) thread pool as your primary EC that backs your ContextShift, and a cached thread pool for jvm-thread-blocking calls, using evalOn to make sure those calls happen on the thread blocking pool
if you do thread-blocking calls without using evalOn (or related) to put it on the right pool, you risk severely hampering the application's efficiency
Victor
@gafiatulin
@/all PR for cross-compilation to 2.13: #54
Jostein Gogstad
@jgogstad
hey guys, any progress on this?
If you would like more maintainers, @gafiatulin and I volunteer
Rolando Manrique
@rolandomanrique
Hi @jgogstad sorry for the delay, its been hard to sync up since Matt is currently the only one with merge access and access to LendUp's GPG keys... I'll try to sync up with him and get this merged and published ASAP. We will also be moving this to a separate GitHub org so we can open it to more maintainers. Once we merge and publish these last PRs we will go ahead and do that.
matthewgraf
@matthewgraf
Starting to merge these, running into a build error. Looks like we need to use openjdk8 instead or oraclejdk8
ah. looks like an easy fix
matthewgraf
@matthewgraf
I got 48 merged, and the master build fixed. I gotta get some other work done atm, but will revisit later this evening/tonight
Sorry again for the delay. Ditto what Rolando said about moving this to another org, we need more maintainers besides me :sweat:
Jostein Gogstad
@jgogstad
no problem, thanks for laying down the effort!
matthewgraf
@matthewgraf
No prob! Now that the build is fixed I should be able to get those other PRs merged today
matthewgraf
@matthewgraf

@gafiatulin would you mind pulling master into your branch? lendup/fs2-blobstore#49

Looks like Travis is using the travis.yml from your branch, not master. I thought it would merge master first and use travis.yml from there. Just trying to get a green build before merging

Victor
@gafiatulin
@matthewgraf #49 is green now
matthewgraf
@matthewgraf
Ok thank you. I'll get to it today
matthewgraf
@matthewgraf
@jgogstad can you fix conflicts on lendup/fs2-blobstore#50 ? I will merge this today. Trying to get this one in and #52 before doing a release
also @gafiatulin for lendup/fs2-blobstore#52
they might be straightforward but you probably have more context
Victor
@gafiatulin
@matthewgraf I fixed #52. It also includes changes from #50
Victor
@gafiatulin
#54 is green now @matthewgraf @rolandomanrique
matthewgraf
@matthewgraf
cool, merged. I'll get that release out now
matthewgraf
@matthewgraf
publishing now, will be released prettttty soon
matthewgraf
@matthewgraf
done. 0.6.0
Jostein Gogstad
@jgogstad
sorry, had some time off for honeymoon
you’re not apart of the fs2-blobstore org, @matthewgraf, added you as collaborator, but I still can’t add you as a reviewer for PRs
can you have a look at fs2-blobstore/fs2-blobstore#2 ?
Doug Clinton
@DougC
Hi. I’ve just put up a PR to update fs2 to 2.2.2. It means dropping scala 2.11 support but brings the fs2 version in line with http4s 0.21.x. lendup/fs2-blobstore#60
Rolando Manrique
@rolandomanrique
@DougC take a look at the repo in the new org fs2-blobstore... https://github.com/fs2-blobstore/fs2-blobstore
@matthewgraf can you add a big not on LendUp repo saying that we moved?
Doug Clinton
@DougC
@rolandomanrique ahh, thanks for the pointer. I was unaware of the move.
matthewgraf
@matthewgraf
@rolandomanrique yeah if I still have access I'll add a note there. Good to see this getting moved since nobody at LendUp maintains this :joy:

Btw has anyone seen this problem when using bufferedPut on an S3Store ?
Failed to reset the request input stream; If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)

I'm wondering if this is a problem specific to our usage of S3Store or an underlying problem with S3Store, involving puts on large files

I saw this brought up on Stackoverflow and elsewhere, I'll dig into it but was just curious to see if anyone else encountered this
Rolando Manrique
@rolandomanrique
Matt I think I saw this once when the size provided in the path did not match the actual size uploaded to s3 (that time happened because we were reading zipped size and uploading unzipped content)
matthewgraf
@matthewgraf
Interesting. I think we might be experiencing this for a different reason. Since we are using bufferedPut I believe the chunk size should be calculated correctly since blobstore is temporarily buffering that data to a local filestore
Maybe I'm misunderstanding bufferedPut. Does it write the entire file to the local filesystem before writing to the output Path?
Also worth noting this flow has been untouched and working for a while - it appears once we are dealing with larger files (e.g. 5 GB in this case) it becomes a problem
matthewgraf
@matthewgraf
Damn, it sounds like we have to write the full file locally before putting to s3, since s3 requires the content size when you initiate the upload. I think your code comment on bufferedPut is essentially saying the same thing
wonder if there is a workaround.. would be nice to stream data into s3 w/o having to write a full file to some other store
Doug Clinton
@DougC
Is this still the right channel for discussion? It’s still associated with the lendup repo. I’m doing some work with the sftp store and just hit a problem listing files that I think is resolved by this fs2-blobstore/fs2-blobstore@23375b1
Would it be possible to push out a release with that change?
Doug Clinton
@DougC
A colleague of mine has just raised this PR fs2-blobstore/fs2-blobstore#9
DeanZi
@DeanZi
Anyone can please share an example of an app to connect to s3?
Hopefully a working code sample that a user can modify his own credentials and access file in his s3, from terminal or workspace.
Jostein Gogstad
@jgogstad
@DeanZi : you configure this on the TransferManager, see the main readme in the repo. E.g.
    val awsCredentials      = new BasicAWSCredentials(accessKey, secretKey)
    val credentialsProvider = new AWSStaticCredentialsProvider(awsCredentials)
    val transferManager = TransferManagerBuilder.standard().withS3Client(AmazonS3ClientBuilder.standard.withCredentials(credentialsProvider).withRegion(region).build) .build
Brian P. Holt
@bpholt
I am right in thinking this comment in the Store[F].put scaladoc is out of date? It looks like S3Store has support for multipart uploads when the size of the upload is unknown.
Victor
@gafiatulin
@bpholt yes, this comment is out of date since fs2-blobstore/fs2-blobstore#40 got merged. Let me update that.
Binh Nguyen
@ngbinh
Hi, is there a snapshot version of blobstore that depends on cats effect 3.0.X?
Aaron Delaplane
@AaronDelaplane
Is there a version of this library that anyone is using with Cats Effect 3 in production?
Aaron Delaplane
@AaronDelaplane
Specifically, is anyone using v0.9.0-beta3 in production?