Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 03:02
    SethTisue synchronize #1345
  • 02:43
    SethTisue synchronize #1345
  • 01:54
    SethTisue synchronize #1345
  • 01:33
    SethTisue commented #9447
  • 01:31
    SethTisue labeled #9447
  • Jan 21 22:33
    SethTisue edited #1345
  • Jan 21 22:32
    SethTisue edited #1345
  • Jan 21 22:31
    SethTisue synchronize #1345
  • Jan 21 21:21
    japgolly commented #9447
  • Jan 21 21:04
    SethTisue synchronize #1345
  • Jan 21 20:55
    SethTisue synchronize #1345
  • Jan 21 20:54
    SethTisue synchronize #1345
  • Jan 21 20:43
    SethTisue synchronize #1345
  • Jan 21 20:27
    SethTisue synchronize #1345
  • Jan 21 20:08
    scala-jenkins milestoned #9447
  • Jan 21 20:08
    lrytz opened #9447
  • Jan 21 20:08
    SethTisue synchronize #1345
  • Jan 21 20:08
    SethTisue synchronize #1345
  • Jan 21 19:58
    SethTisue synchronize #1345
  • Jan 21 19:56
    SethTisue synchronize #1345
Luis Miguel Mejía Suárez
@BalmungSan
@DCameronMauch you are in a point where pure FP can help.
If it is worth or not is up to you.
If you are not using things like IO yet, I would not introudce them just for this.
Rob Norris
@tpolecat
Side-effecting code doesn't compose so I think it's best to write it top-down and make the execution path crystal clear.
Luis Miguel Mejía Suárez
@BalmungSan
Although, not sure if Eval would be enough to safely use things like whenA?
Raitis Veinbahs
@siers
I want to have a default method in my trait over F[_] that would return pure in some instance, which requires Applicative, which is not possible.
Should I move the responsibility of writing default code to each extending class?
D Cameron Mauch
@DCameronMauch
I really liked the Option orElse solution
Is there any way to do that while keeping intermediate values in scope?
@Ichoran that pattern also doesn’t work for the same issue of intermediate values being in scope
Luis Miguel Mejía Suárez
@BalmungSan
@siers I would say yes, especially if you want to keep your trait clean from dependencies.
Depending on what else you have on that trait, you may add a defualt constructor on the companion object.
D Cameron Mauch
@DCameronMauch
For example, if I had to computer something in the “bad_thing” block, and refer to it in the “other_bad_thing” conditional...
Luis Miguel Mejía Suárez
@BalmungSan
@DCameronMauch I really do not understand why you want to change your code?
D Cameron Mauch
@DCameronMauch
Before, it wasn’t compiling
because of the warning about using returns
Luis Miguel Mejía Suárez
@BalmungSan
Yeah because of the realy return.
You can fix that with the pattern match.
D Cameron Mauch
@DCameronMauch
no idea what changed to let it compile now
Luis Miguel Mejía Suárez
@BalmungSan
(fromCount, toCount) match {
  case (None, None) =>
    logger.error(s"error processing $from: could not validate counts")
    false

  case (Some(from), Some(to)) if (form != to) =>
    logger.error(s"error processing $from: counts do not match: ${from}, ${to}")

  case _ =>
    logger.info(s"creating _SUCCESS file for $to")
    destination.s3.putObject(bucket, to + "_SUCCESS", "")
    true
}
(I would prefix the Option values with opt)
Raitis Veinbahs
@siers
@BalmungSan Alright! How would a default constructor look like roughly?
Luis Miguel Mejía Suárez
@BalmungSan
@siers as I said, it depends on what else the trait provides for it to make sense.
Can you create a little version of your typeclass and the default method you want to add?
Raitis Veinbahs
@siers
@BalmungSan Ah, I found what I need already. My colleagues solved this in other parts of the code with an abstract class, which can be parametrized over F with type bounds.
Luis Miguel Mejía Suárez
@BalmungSan
Ah.
I thought the problem was not wanting to require an Applicative.
Luis Miguel Mejía Suárez
@BalmungSan
@mariacris431 @FunctionalUs_twitter please post job offers in (and only in) the Scala job board.
Fabio Labella
@SystemFw
this one isn't even a scala one
Ichoran
@Ichoran
@BalmungSan - See, that's exactly why I didn't want to use a match...the logic there is wrong. If either one is None then it's an error. (The same error.)
Luis Miguel Mejía Suárez
@BalmungSan
Ah I missed the or, I saw an and.
See, that is why I do not like ifs :stuck_out_tongue:
Also, my bad for trying to mimic the original code.
I would probably start with the good case and then start adding each error.
Ichoran
@Ichoran
Yeah, you often have different natural orders when using if-statements vs. matches
Luis Miguel Mejía Suárez
@BalmungSan
:+1:
Alexis Hernandez
@AlexITC
FYI: There is an ongoing discussion about Scala vs Kotlin https://news.ycombinator.com/item?id=25843390
Ichoran
@Ichoran
@DCameronMauch -- By the way, I'd write your code more like this. Except I would use my Ok instead of Either, and I have a macro called .? that does the match { case Left(e) => return Left(e) ... stuff.
def tryCount(s: String, from: String, who: String): Either[String, String] = 
  try {
    (s !!).split("\n").find(_.contains("Feature Count")) match {
      case Some(c) => Right(c)
      case _       => Left(s"error processing $from: could not validate $who")
    }
  }
  catch {
    case t if NonFatal(t) =>
      Left(s"Error getting $who while proccessing $from: ${t.getMessage}")
  }


def convertShapeFileImpl(
  destination: S3Destination, from: String, to: String, bucket: String
): Either[String, String] = {
    val vsiFrom = s"/vsis3_streaming/${bucket}/${from}"
    val vsiTo = s"/vsis3/${bucket}/${to}"

    val result = s"ogr2ogr -dim 2 -f GeoJSON $vsiTo $vsiFrom" !

    if (result != 0) return Left("error processing " + from)

    val fromCount = tryCount(s"ogrinfo -ro -al -so $vsiFrom", from, "fromCount") match {
      case Left(e) => return Left(e)
      case Right(x) => x
    }
    val toCount   = tryCount(s"ogrinfo -ro -al -so $vsiTo",   to,   "toCount") match {
      case Left(e) => return Left(e)
      case Right(x) => x
    }

    if (fromCount != toCount.get)
      Left(s"error processing $from: counts do not match: ${fromCount.get}, ${toCount.get}")
    else 
      Right(s"creating _SUCCESS file for $to")
}

def convertShapeFile(destination: S3Destination, from: String, to: String) =
  convertShapeFileImpl(destination, from, to, destination.bucket) match {
    case Left(e) =>
      logger.error(e)
      false
    case Right(x) =>
      logger.info(x)
      destination.s3.putObject(destination.bucket, to + "_SUCCESS", "")
  }
So it'd just look like val toCount = tryCount(s"ogrinfo -ro -al -so $vsiTo", to, "toCount").?
oybek
@oybek

Hello!
I'm having troubles figuring out this task:
I got a Seq[Int], and function Int => Future[Int], I want to process every number in seq using this function, but at the same time only batchSize future must be spawned, I came out with this code:

object Solution extends App {
  def batchTraverse(seq: Seq[Int], func: Int => Future[Int], batchSize: Int): Future[Seq[Int]] = {
    seq
      .grouped(batchSize)
      .foldLeft(Future.successful(Seq.empty[Int])) {
        case (acc, curBatch) =>
          for {
            x <- acc
            y <- Future.traverse(curBatch)(func)
          } yield x ++ y
      }
  }

  def func(i: Int): Future[Int] = Future {
    println(s"running func($i)")
    Thread.sleep(2000)
    i * 100
  }

  val seq = 1 to 12
  println(
    Await.result(batchTraverse(seq, func, 4), 100.seconds)
  )
}

I'm expecting this:

running func(1)
running func(2)
running func(3)
running func(4)
~2 sec delay
running func(5)
running func(6)
running func(7)
running func(8)
~2 sec delay
...

But getting:

running func(4)
~2 sec delay
running func(3)
~2 sec delay
running func(2)
~2 sec delay
running func(1)
~2 sec delay
running func(8)
~2 sec delay
...

why is it working in this way?
When I'm doing just:
Await.result(Future.traverse(1 to 4)(func), ...) all futures are working parallel

Luis Miguel Mejía Suárez
@BalmungSan
@oybek The best way is to just traverse everything with a custom ExecutionContext that ensures only batchsize concurrent tasks.
Also, in your example, the sleep is per task not per batch.
oybek
@oybek

@oybek The best way is to just traverse everything with a custom ExecutionContext that ensures only batchsize concurrent tasks.

agree, but it is the different problem, of course you can limit threadpool size
but I want to solve this problem without underhood tools

Luis Miguel Mejía Suárez
@BalmungSan
WDYM with underhood tools?
oybek
@oybek

WDYM with underhood tools?

I mean that by the problem definition it is restricted to change parameters of executionContext

Luis Miguel Mejía Suárez
@BalmungSan
@oybek Your solution seems to work, at least in scastie: https://scastie.scala-lang.org/BalmungSan/w8CIKlbaRCemcoG39HfWiQ/1
You can not expect each batch to also execute sequentiality inside it.
Also, in the end, this is also governed by the underlying execution context.
So nothing ensures you that each batch will have exactly that number of concurrent tasks.

I mean that by the problem definition it is restricted to change parameters of executionContext

Your problem, as you stated, is to control the number of concurrent tasks... that is the job of an execution context, I really do not see the point in solve that in another way.

oybek
@oybek
I got the solution, Thx @BalmungSan ))
The problem was that every future was executed in single thread))
I created custom threadpool with 100 threads and all worked as I expected
Seth Tisue
@SethTisue
:+1: that's much more in line with the design intent behind Future, it's core to the concept of Future that it's fine to make a zillion of them
Luis Miguel Mejía Suárez
@BalmungSan
Not sure to which you refer @SethTisue I am interested in getting a better understanding of Future for when someone ask a question.