Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 11 01:27
    scala-steward closed #941
  • Aug 11 01:27
    scala-steward opened #942
  • Aug 06 14:42
    scala-steward closed #939
  • Aug 06 14:42
    scala-steward opened #941
  • Aug 06 14:42
    scala-steward closed #911
  • Aug 06 14:42
    scala-steward opened #940
  • Aug 04 15:43
    scala-steward closed #938
  • Aug 04 15:43
    scala-steward opened #939
  • Aug 02 15:52
    scala-steward closed #936
  • Aug 02 15:52
    scala-steward opened #938
  • Aug 02 15:52
    scala-steward opened #937
  • Jul 29 17:25
    scala-steward closed #933
  • Jul 29 17:25
    scala-steward opened #936
  • Jul 29 17:25
    scala-steward opened #935
  • Jul 29 17:25
    scala-steward closed #923
  • Jul 29 17:25
    scala-steward opened #934
  • Jul 23 15:15
    scala-steward closed #932
  • Jul 23 15:15
    scala-steward opened #933
  • Jul 21 15:35
    scala-steward closed #931
  • Jul 21 15:35
    scala-steward opened #932
TapanVaishnav
@TapanVaishnav
Hello all, thank you accepting me on this channel.
I am currently working on good-first-issues.
Also, I like to create a monix-connect for elasticsearch and flink, what do you guys think of the idea?
Pau Alarcón
@paualarco
Welcome @TapanVaishnav :-)
Having those would be amazing! Specially the elasticsearch one, since many companies use it nowadays.
Pau Alarcón
@paualarco
@/all pipeline is back to normal! :D
Pau Alarcón
@paualarco
the reason was that the localstack image was set to latest and they released a breaking changes (but only for some machines) apparently the one that runs our pipeline on github actions was affected.
Alexandru Nedelcu
@alexandru
Actually it's this one: monix/monix-connect@eaf733e
It was meant to be a PR, but apparently I'm bad at using GitHub's interface.
Pau Alarcón
@paualarco
@alexandru no problem, unfortunately domain changes haven't applied
will roll it back :P
Alexandru Nedelcu
@alexandru
@paualarco I think it would need to finish the build or something
In the settings file, it gives a warning that the CNAME file doesn't have a correct format: https://github.com/monix/monix-connect/settings
This means the GitHub Pages haven't been deployed yet (an assumption).
Pau Alarcón
@paualarco
Mmh okay, let's give it some time then, fingers crossed 🤞
Alexandru Nedelcu
@alexandru
Are you fine with the domain change? I was meaning to ask in the PR 😅
I mean it's a proposal. If you don't like it you can revert.
Pau Alarcón
@paualarco
I like it! Thanks for helping with this :)
Alexandru Nedelcu
@alexandru
Is the API published along with the website? As I'm not seeing it in gh-pages: https://github.com/monix/monix-connect/tree/gh-pages
Pau Alarcón
@paualarco
nice!
no, api docs are not there yet monix/monix-connect#149
Piotr Gawryś
@Avasil
What do you think about creating a new module in monix-connect for I/O operations, something like akka-stream-alpakka-file or fs2 io? There's a contributor that is implementing gzip for Observable for STTP and we're wondering where to put it :)
Pau Alarcón
@paualarco
@Avasil sure! the google-cloud-storage connector is already exposing some methods for reading and writing from files which we could abstract in a different module
Pau Alarcón
@paualarco
though it might overlap a bit with monix-nio
Piotr Gawryś
@Avasil
@ghostbuster91 raised a good point to me that compression is not necessarily about files, or I/O so I think we could add it to monix-reactive, fs2 does something similar (text package is in core)
Pau Alarcón
@paualarco
yup, that's right :)
Pau Alarcón
@paualarco
have created monix/monix-connect#321 for bringing downloadMultipart, existsBucket, existsObject, copyObject and an enhanced listObjects operations, will leave it there for a couple of days :)
Tyler M. Kontra
@ttymck
Hi @paualarco just wanted to drop in to say thanks for answering my questions so far, and I hope I can help out with this project :smile:
Pau Alarcón
@paualarco
@ttymck no problem, thank you too for helping :)
Thijs Broersen
@ThijsBroersen
I found an issue in S3 adapter and thought I could make an easy fix and do a PR but I cannot get started, aws-auth does not compile ... getting [error] ## Exception when compiling 6 sources to /home/thijs/repositories/monix-connect/aws-auth/target/scala-2.13/classes [error] java.lang.StackOverflowError [error] scala.tools.nsc.typechecker.Typers$Typer.typedBlock(Typers.scala:2571) [error] scala.tools.nsc.typechecker.Typers$Typer.typedOutsidePatternMode$1(Typers.scala:5911) [error] scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5946) [error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5993) [error] scala.tools.nsc.typechecker.Typers$Typer.typedTyped$1(Typers.scala:5703) [error] scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5939) [error] scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5993) [error] scala.tools.nsc.typechecker.Macros$DefMacroExpander.$anonfun$onSuccess$1(Macros.scala:631) [error] scala.tools.nsc.typechecker.Macros$DefMacroExpander.typecheck$1(Macros.scala:631) [error] scala.tools.nsc.typechecker.Macros$DefMacroExpander.onSuccess(Macros.scala:643) [error] scala.tools.nsc.typechecker.Macros$MacroExpander.$anonfun$expand$1(Macros.scala:582) [error] scala.tools.nsc.Global.withInfoLevel(Global.scala:228)

wanted to adjust this:

def deleteObject(
    bucket: String,
    key: String,
    bypassGovernanceRetention: Option[Boolean] = None,
    mfa: Option[String] = None,
    requestPayer: Option[String] = None,
    versionId: Option[String] = None)(implicit s3AsyncClient: S3AsyncClient): Task[DeleteObjectResponse] = {
    val request: DeleteObjectRequest =
      S3RequestBuilder.deleteObject(bucket, key, bypassGovernanceRetention, mfa, requestPayer, versionId)
    S3RequestBuilder.deleteObject(bucket, key, bypassGovernanceRetention, mfa, requestPayer, versionId)
    deleteObject(request)
  }

into

def deleteObject(
    bucket: String,
    key: String,
    bypassGovernanceRetention: Option[Boolean] = None,
    mfa: Option[String] = None,
    requestPayer: Option[String] = None,
    versionId: Option[String] = None): Task[DeleteObjectResponse] = {
    val request: DeleteObjectRequest =
      S3RequestBuilder.deleteObject(bucket, key, bypassGovernanceRetention, mfa, requestPayer, versionId)
    deleteObject(request)
  }
Pau Alarcón
@paualarco
@ThijsBroersen thanks for reporting that! I realised about that last week but didn't changed yet... though it does not alter the functionality of the method
Pau Alarcón
@paualarco
compiling that submodule should be fine, maybe that's something to do with your local config? have seen a similar where it seems like they fixed it by increasing -Xss8m.
Thijs Broersen
@ThijsBroersen
I already did s3/compile ... did not work.
regarding the change ... notice that I also removed the implicit s3-client as it is already part of the S3 trait and was not used by the method implementation
Pau Alarcón
@paualarco
have you changed the SBT_OPS before?
maybe first try just compiling the aws-auth subproject?
sbt -mem 2048 aws-auth/compile
Thijs Broersen
@ThijsBroersen
that works, thanks!
didn't find this tip in the monix-connect documentation :P
Pau Alarcón
@paualarco
no problem, that's a handy one :)
Jenda Kolena
@jendakol

Hi guys!
I've just found this library and it's a nice piece of work! I have several ideas for improvement though - it might be better to discuss it here than in GH issue I guess... Everything is related to S3 connector as it's what I want to use.

  1. It'd be great if there's a factory method taking already parsed MonixAwsConf so one can fill it manually (or parse it, whatever) and not has to use fromConfig.
  2. It'd be great if pureconfig readers are configurable regarding casing. We're used to have camelCase and existing readers for your case classes are kebab-case only.
  3. It'd be great if fromConfig takes a ConfigSource so users are not forced to have the configuration in a hardcoded place (root, in this case).
  4. Why are config classes (e.g. MonixAwsConf) private? :-D
  5. I'd love to see offset or range parameter in the downloadMultipart method - we need to be able to resume download of a big file and that is currently not possible.

So, what do you think? I'm willing to help implementing these changes, I just want your approval first so I don't waste my time :-D
Thanks!

Pau Alarcón
@paualarco
Hi @jendakol
Appreciate the feedback! :D I basically agree in all the points you have raised and would be happy to adopt them.
I was already aware of 1 and 2 , though I have had no time to address it, yet.
In response to 4, it is private because the current signatures does not allow to overwrite nor read it, so we will make it public access if that changes.
We could create issues to keep track of them, would you be interested in contributing to those?
Pau Alarcón
@paualarco
@jendakol have addressed 1, 2 and 3, 4 in monix/monix-connect#685, would appreciate your review/feedback if you have time :)
cwgroppe
@cwgroppe

@here greetings everyone, i'm very new to Fargate/ECS/Batch so forgive my ignorance, but I have a docker image/container thats running a scala batch job. it uses the aws-sdk-v2, hadoop client, and monix to make a bunch of parallel api calls to an http endpoint, bring the data back, do some transforms, and then make a bunch of parallel writes to s3. everything works fine locally, but when i run it as a batch job with fargate i get this error:

java.nio.file.AccessDeniedException: ...
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by ContainerCredentialsProvider : com.amazonaws.AmazonServiceException: Too Many Requests (Service: null; Status Code: 429; Error Code: null; Request ID: null)
when i try to write the data to s3 as parquet files with code that looks very similar to the example from the docs :

val parquetWriter: ParquetWriter[GenericRecord] = {
AvroParquetWriter.builderGenericRecord
.withConf(conf)
.withSchema(schema)
.build()
}

// returns the number of written records
val t: Task[Long] = {
Observable
.fromIterable(elements) // Observable[Person]
.map( => personToGenericRecord())
.consumeWith(ParquetSink.fromWriterUnsafe(parquetWriter))
}
it doesnt happen with every write taking place in the job, but it does with a lot of them. Here are my hadoop settings if that helps:

conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")
conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
conf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.ContainerCredentialsProvider")

The reading ive done has made me more confused about the whole thing, like maybe its container settings for using IAM for creds, but it half works as is? i dunno,
I would just like this process to run like it does in docker on my local machine, any help would be much appreciated.

Pau Alarcón
@paualarco
hi @cwgroppe :)
are you using the parquet or hadoop connectors?
have you considered replacing the fs.s3a.aws.credentials.providerby org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider?
It might also be a good idea to ask these same question to the apache hadoop community
cwgroppe
@cwgroppe
@paualarco i think im using the parquet connector... whats SimpleAWSCredentialsProvider vs ContainerCredentialsProvider? ... and heard on hadoop community, i will.
also sorry for the late response... been dealing with some stuff and havent been near a comp for about a week
cwgroppe
@cwgroppe
@paualarco looking at the hadoop docs under SimpleAWSCredentialsProvider - ive been using the IAM roles to get creds while running this thing in my company's VPC and i dont think SimpleAWSCredentialsProvider allows for this. could be me misunderstanding though
Pau Alarcón
@paualarco
Just realized that I did not respond @cwgroppe, did you resolved or gave up on it?
I belive that the fact of using IAM roles differs from SimpleAwsCreds, so you should still use the second one if using org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, otherwise u might have had to use Default configuration, which would fallback to using the IAM roles.
Sorry for the late response