Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 00:40

    mergify[bot] on scribe-3.11.0

    (compare)

  • 00:40

    mergify[bot] on master

    Update scribe to 3.11.0 Merge pull request #1729 from s… (compare)

  • 00:40
    mergify[bot] closed #1729
  • 00:38

    mergify[bot] on fs2-core-3.6.0

    (compare)

  • 00:38

    mergify[bot] on master

    Update fs2-core, fs2-reactive-s… Merge pull request #1728 from s… (compare)

  • 00:38
    mergify[bot] closed #1728
  • 00:24
    mergify[bot] labeled #1729
  • 00:24
    mergify[bot] labeled #1728
  • 00:23
    softwaremill-ci opened #1729
  • 00:23

    softwaremill-ci on scribe-3.11.0

    Update scribe to 3.11.0 (compare)

  • 00:23
    softwaremill-ci opened #1728
  • 00:23

    softwaremill-ci on fs2-core-3.6.0

    Update fs2-core, fs2-reactive-s… (compare)

  • Feb 08 22:36
    kvma opened #1727
  • Feb 08 10:14
    adamw synchronize #1703
  • Feb 08 00:26
    mergify[bot] labeled #1726
  • Feb 08 00:26
    softwaremill-ci opened #1726
  • Feb 08 00:25

    softwaremill-ci on zio-2.0.7

    Update zio, zio-streams to 2.0.7 (compare)

  • Feb 07 19:22

    adamw on v3.8.11

    (compare)

  • Feb 07 19:22

    adamw on master

    Release 3.8.11 (compare)

  • Feb 07 15:49

    mergify[bot] on zio-nio-2.0.1

    (compare)

Vitalii Supryhan
@Supryhan
image.png
Brandon Hudgeons
@bhudgeons
I'm using the ZIO backend and trying to diagnose a memory leak. threadLocalMap of io.netty.util.concurrent.FastThreadLocalThread #3009 [Thread, Stack Local] "AsyncHttpClient-5-1" tid=196 [RUNNABLE] Native ID: 16561879 14856 384 is holding on to PoolChunks containing the byte buffers of responses. Any suggestions?
Brandon Hudgeons
@bhudgeons
ah ... it looks like just using the AsyncHttpClientZioBackend().flatMap { implicit backend => as recommended in the docs creates a client every time?
10 replies
Ibrahim Moufti
@Mouftizo

Hello everyone : )

Apologies in advance if the question does not follow guidelines as I am new here. I am also new to sttp and Monix, and I am trying to learn more about these libraries. My goal is to fetch data (client-side) from a given API via HTTP GET requests -> parse JSON responses -> write this information to a database. My question pertains to the first part only. My objective is to run get requests in an asynchronous (hopefully fast) way while having a way to either avoid or handle rate limits.

Below is a snippet of what I have already tried, and seems to work for a single request:

    package com.github.me.client

    import io.circe.{Decoder, HCursor}
    import sttp.client._
    import sttp.client.circe._
    import sttp.client.asynchttpclient.monix._
    import monix.eval.Task

    object SO extends App {

      case class Bla(paging: Int)

      implicit val dataDecoder: Decoder[Bla] = (hCursor: HCursor) => {
        for {
          next_page <- hCursor.downField("foo").downArray.downField("bar").as[Int]
        } yield Bla(next_page)
      }

      val postTask = AsyncHttpClientMonixBackend().flatMap { implicit backend =>
        val r = basicRequest
          .get(uri"https://foo.bar.io/v1/baz")
          .header("accept", "application/json")
          .header("Authorization", "hushh!")
          .response(asJson[Bla])

        r.send() // How can I instead of operating on a single request, operate on multiple
          .flatMap { response =>
            Task(response.body)
          }
          .guarantee(backend.close())
      } 

      import monix.execution.Scheduler.Implicits.global

      postTask.runSyncUnsafe() match {
        case Left(error) => println(s"Error when executing request: $error")
        case Right(data) => println(data)
      }
    }

My questions:

  1. How can I operate on several GET Requests (instead of a single request) while keeping the code asynchronous and composable
  2. How can I avoid or handle hitting rate-limits imposed by the api server

On a side note, I am not particularly fixated on using Monix, and I am open to any other back-end or implementation

16 replies
sebver
@sebver

Hi! This is a very small thing, but would you be open to a PR that adds a type alias for a zio SttpBackend (the part within the Has)
https://github.com/softwaremill/sttp/blob/1e34bff7ef8bb0200d456bc8f268b3b25fdfa6ce/async-http-client-backend/zio/src/main/scala/sttp/client/asynchttpclient/zio/package.scala#L13

I find myself redefining it in several codebases to make it a bit shorter when I'm passing it around.

1 reply
vonchav
@voonchav_gitlab
@adamw Should sttp-client be using sttp-model 1.1.4? It seems even the latest version sttp-client 2.2.4 is still using sttp-model 1.1.3 whereas Tapir is already using 1.1.4? :)
2 replies
Ibrahim Moufti
@Mouftizo

Hello everyone : )

I am currently playing around with STTP using the Monix backend. I am mainly stuck with closing the backend after all my requests (each request is a task) have been processed.

I have created sample code to resemble my issue (in case that helps answering my question):

import sttp.client.asynchttpclient.monix._
import monix.eval.Task
import monix.reactive.Observable
import sttp.client.{Response, UriContext}

import scala.concurrent.duration.DurationInt

object ObservableTest extends App {

  val activities = AsyncHttpClientMonixBackend().flatMap { implicit backend =>
    val ids: Task[List[Int]] = Task { (1 to 3).toList }
    val f: String => Task[Response[Either[String, String]]] = (i: String) => fetch(uri"$i", "")
    val data: Task[List[Task[Response[Either[String, String]]]]] = ids map (_ map (_ => f("https://heloooo.free.beeceptor.com/my/api/path")))
    data.guarantee(backend.close()) // If I close the backend here, I can't make generate requests after
    // I have attempted to return a Task containing a tuple of (data, backend) but closing the backend from outside of the scope did not work as I expected
  }
  import monix.execution.Scheduler.Implicits.global
  val obs = Observable
    .fromTask(activities)
    .flatMap { listOfFetches =>
      Observable.fromIterable(listOfFetches)
    }
    .throttle(3 second, 1)
    .map(_.runToFuture)

  obs.subscribe()
}

And my fetch (api call maker) function looks like:

  def fetch(uri: Uri, auth: String)(implicit
      backend: SttpBackend[Task, Observable[ByteBuffer], WebSocketHandler]
  ) = {
    println(uri)
    val task = basicRequest
      .get(uri)
      .header("accept", "application/json")
      .header("Authorization", auth)
      .response(asString)
      .send()

    task
  }

As my main task contains other tasks, which I later need to process, I need to find an alternative way to close the Monix backend from outside. Is there a clean way to do close the backend after I consume the requests in List[Task[Response[Either[String, String]]]]?

3 replies
bond15
@bond15

Hello!
Is the OpenAPI scala-sttp generator usable? I ask because It's status is listed in beta (https://openapi-generator.tech/docs/generators) and I'm getting serialization errors on simple string body responses.

 def getHealth(): ApiRequestT[String] =
    basicRequest
      .method(Method.GET, uri"$baseUrl/health")
      .contentType("application/json")
      .response(asJson[String])
import org.openapitools.client.api.DefaultApi
import org.openapitools.client.core.ApiInvoker._
import org.openapitools.client.core.SttpSerializer
import sttp.client.HttpURLConnectionBackend

implicit val s = new SttpSerializer()
implicit val b = HttpURLConnectionBackend()

val api = DefaultApi(<my string url>)

api.getHealth().result

results in

org.openapitools.client.core.HttpException: [200] OK: Running...

The body is correct but a serialization error is thrown.

9 replies
ex0ns
@ex0ns

Hello ! I'm trying to use sttp with ZIO backend (2.2.4)but I end up having a runtime exception

─An unchecked error was produced.
║ java.lang.IncompatibleClassChangeError: Found interface zio.ZQueue, but class was expected
║       at zio.interop.reactivestreams.Adapters$.process(Adapters.scala:71)
║       at zio.interop.reactivestreams.Adapters$.$anonfun$publisherToStream$6(Adapters.scala:49)
║       at zio.ZManaged.$anonfun$flatMap$6(ZManaged.scala:315)
║       at zio.internal.FiberContext.evaluateNow(FiberContext.scala:815)
║       at zio.internal.FiberContext.$anonfun$evaluateLater$1(FiberContext.scala:687)
║       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
║       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
║       at java.lang.Thread.run(Thread.java:748)

I can't figure out why this is happening, does someone have any clue ? I unfortunately can't share the code but it's a really simple POST request I'm trying to make, followed by a Runtime.default.unsafeRunToFuture

1 reply
Esenbaev Nursultan
@eanea

Hello, everyone!
Got this error

value mapK is not a member of sttp.client.SttpBackend

snippet:
https://scastie.scala-lang.org/OgnPGIorTfGep6rj3wfl6A

I've imported import sttp.client.impl.cats.implicits._
but it doesn't help.

7 replies
Adam Warski
@adamw
hey :) we've released sttp-client 3.0.0-RC1, feedback or further suggestions on improvements would be most welcome! https://blog.softwaremill.com/whats-coming-up-in-sttp-client-3-30d01ab42d1b
2 replies
Kamil Kloch
@kamilkloch
Hello all, how to convert org.scalajs.dom.raw.File to sttp.client.dom.experimental.File? In other words, I would like to use a platform-agnostic method upload(file: SttpFile) and on the JS side all I have is a org.scalajs.dom.raw.File. Thanks!
Kamil Kloch
@kamilkloch
This ugly workaround works, any other ideas? SttpFile.fromDomFile(f.asInstanceOf[sttp.client.dom.experimental.File])
2 replies
Yann Moisan
@YannMoisan

Hi. I'm migrating from sttp v1 to sttp v2. I use a tagless final approach. In order to map on a Response, I was using

implicit val monad: MonadError[R] = backend.responseMonad

in v2, it doesn't work anymore.
Do I have to declare a context bound instead R[_]: Functor

6 replies
Yann Moisan
@YannMoisan

Hello. I've seen the logging backend in sttp v2. It's always nice to be able to remove some boilerplate in my code.

How can I log also the body of a response ?

The simplest way I found is to extend the Slf4jLoggingListener :

(I need to be in this package because Logger is private)

package sttp.client.logging.slf4j

import sttp.client.listener.ListenerBackend
import sttp.client.logging.LogMessages.requestToString
import sttp.client.{Identity, Request, Response, SttpBackend}

object CustomSlf4jLoggingListener {
  private val logger = new Logger("sttp.client.logging.slf4j.Slf4jLoggingBackend")

  def apply[F[_], S, WS_HANDLER[_]](
      delegate: SttpBackend[F, S, WS_HANDLER]
  ): SttpBackend[F, S, WS_HANDLER] =
    ListenerBackend.lift(delegate, new CustomSlf4jLoggingListener(logger))
}

class CustomSlf4jLoggingListener(logger: Logger) extends Slf4jLoggingListener(logger) {
  override def requestSuccessful(
      request: Request[_, _],
      response: Response[_],
      tag: Unit
  ): Identity[Unit] =
    if (response.isSuccess) {
      logger.debug(response_(request, response))
    } else {
      logger.warn(response_(request, response))
    }

  def response_(request: Request[_, _], response: Response[_]): String =
    s"For request: ${requestToString(request)}, got response: ${responseToString(response)}"

  def responseToString(response: Response[_]): String =
    s"${response.code}, body: ${response.body}"
}
2 replies
Harsh
@frog_splash_twitter

Hi All, I am new to sttp and trying to learn few basics of it.
In one of the issues I am facing I have not been getting right json response for this code

val baseRequest: RequestT[Empty, Either[String, String], Nothing] = basicRequest.header("accept", "application/vnd.github.v3+json")
    val url: Uri = uri"https://api.github.com/events"

    sttpBackend
      .send {
        baseRequest
          .readTimeout(config.http.timeout)
          .get(url)
          .response(asJson[Either[GitErrorResponse, List[CommitEvent]]])
      }
      .map[Either[Error, List[CommitEvent]]] {
        response: Response[Either[ResponseError[CirceError], Either[GitErrorResponse, List[CommitEvent]]]] =>
          response.translate
      }
      .handleError { throwable: Throwable =>
        throwable.translate
      }

compare that to this basic scala code its working as expected

val url = "https://api.github.com/events"
  val result = scala.io.Source.fromURL(url).mkString
  println(result)

Can someone help ?

1 reply
barthorre
@barthorre
Hi everyone, anyway here has some experience with opening a websocket connection with a custom protocol?
15 replies
kbakha
@kbakha
Hi there. Is there any configuration for connection pools for okhttp backend?
2 replies
Łukasz Drygała
@ldrygala
Hi everyone, is there any way to convert org.scalajs.dom.File into sttp.client.dom.experimental.File. I wanted to send a file selected from UI.
11 replies
Glen Marchesani
@fizzy33
@adamw is there a place that talks about the differences between v2 and v3 aka the what's new ? thanks
4 replies
Glen Marchesani
@fizzy33
also thank you, thank you, thank you for having a new package name for v3 (other projects that didn't do that have caused my organization a fair bit of chaos) so it is appreciated.
Glen Marchesani
@fizzy33
Has anyone gotten sttp working with sse / server sent events ? I plan to use the streaming support to do it
3 replies
This message was deleted
Mark Hammons
@markehammons
Is there ever a situation where sttp will attempt to reread a stream body?
19 replies
vonchav
@voonchav_gitlab
Hi @adamw, I'm getting around to finally try out sttp3 and I'm having problem with migrating fromMetadata to the new signature. Can you provide an example for the new signature?
v2's signature is: def fromMetadata[T, S](f: ResponseMetadata => ResponseAs[T, S]): ResponseAs[T, S]
v3's is: def fromMetadata[T, R](default: ResponseAs[T, R], conditions: ConditionalResponseAs[T, R]*)
Thanks!
14 replies
Aleksandar Skrbic
@aleksandarskrbic
    val request = basicRequest
      .body(authPayload)
      .post(uri"${config.url}")
      .response(asJson[AccessToken])

    retry.Backoff(5, 1.second).apply(request.send())
4th line throws this error
java.lang.NoSuchMethodError: sttp.client.ResponseAs$.deserializeRightWithError(Lscala/Function1;)Lscala/Function1;
Does anyone know what could be a problem? I have implicit Decoder for AccessToken
I'm using AkkaHttpBackend
Michael
@grouzen
Hi.
@adamw why tapir 0.17.0-M2 brings sttp3 with package name sttp.client3 when sync-http-client-backend-cats 3.0.0-RC3 uses sttp3 with package name sttp.client? How should I compose them? ;)
Michael
@grouzen
In the end I have such an error:
inferred type arguments [mypackage.AppEffect,sttp.client.SttpBackend[mypackage.AppEffect,Any]] do not conform to method fromEffect's type parameter bounds [F[_],I <: sttp.client3.SttpBackend[mypackage.AppEffect,Any]]
2 replies
Aleksandar Skrbic
@aleksandarskrbic

java.lang.NoSuchMethodError: sttp.client.ResponseAs$.deserializeRightWithError(Lscala/Function1;)Lscala/Function1;

I've managed to solve this. I had different versions of sttp and sttp-akka-http backend so that was the problem :)

Glen Marchesani
@fizzy33
so I am getting EventSource (aka server sent events) working in scala js and scala jvm. We had a bunch of weird issues where the scala js code worked but not as expected (things like EventStream in chrome wasn't populated). That Chrome symptom says that you are not using a polyfill and not EventSource directly. We are like but we are using EventSource. The end of the story is it turns out we were using our scala jvm based server sent events parser with sttp's monix fetch backed in the brower and it just worked...
1 reply
nobody noticed this for a long long time
pool
@hamstakilla
Hello, im getting this weird compilation errors while moving from POM to SBT(sttp v. 1.7.2)
"No implicits found for parameter handler: SttpHandler[R_, Nothing]"
I have sync backend implicit in scope. Maybe i'm missing some compiler flag or something?
2 replies
Tobias Roeser
@lefou
Hello, can anybody tell me where the * in class AkkaHttpBackend extends SttpBackend[Future, Source[ByteString, Any], Flow[Message, Message, *]] comes from?
6 replies
IJ IDEA can't
Denis Mikhaylov
@notxcain
Hi! Does anyone know any sttp benchmarks with backend comparison? Or maybe someone compared Java 11 HttpClient with Netty?
Glen Marchesani
@fizzy33
is there something I need to do to cleanup resources on errors ? I have a sttp client request that is failing because the server isn't started after a little bit I get a java.io.IOException: Too many open files and at that point it never recovers
this is in an isolated use case so there is nothing else using up descriptors
the full stack trace is
17:58:15.191 | ERROR | scala-execution-context-global-14 | a8.hermes.ServerSentEventsFactory. | error with stream will restart @ Some(30)
java.lang.IllegalStateException: failed to create a child event loop
  at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
  at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
  at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
  at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
  at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86)
  at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81)
  at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68)
  at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:32)
  at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:21)
  at org.asynchttpclient.netty.channel.ChannelManager.<init>(ChannelManager.java:133)
  at org.asynchttpclient.DefaultAsyncHttpClient.<init>(DefaultAsyncHttpClient.java:92)
  at sttp.client.asynchttpclient.AsyncHttpClientBackend$.defaultClient(AsyncHttpClientBackend.scala:380)
  at sttp.client.asynchttpclient.monix.AsyncHttpClientMonixBackend$.$anonfun$apply$1(AsyncHttpClientMonixBackend.scala:76)
  at monix.eval.internal.TaskRunLoop$.startLight(TaskRunLoop.scala:271)
  at monix.eval.Task.runAsyncOptF(Task.scala:812)
  at monix.eval.Task.runAsyncOpt(Task.scala:710)
  at monix.eval.Task.runAsync(Task.scala:660)
  at monix.reactive.internal.builders.TaskAsObservable.unsafeSubscribeFn(TaskAsObservable.scala:30)
  at monix.reactive.internal.operators.ConcatMapObservable.unsafeSubscribeFn(ConcatMapObservable.scala:76)
  at monix.reactive.internal.operators.OnErrorRecoverWithObservable.unsafeSubscribeFn(OnErrorRecoverWithObservable.scala:35)
  at monix.reactive.internal.operators.LiftByOperatorObservable.unsafeSubscribeFn(LiftByOperatorObservable.scala:30)
  at monix.reactive.observables.ChainedObservable$.$anonfun$subscribe$1(ChainedObservable.scala:74)
  at monix.execution.internal.Trampoline.monix$execution$internal$Trampoline$$immediateLoop(Trampoline.scala:66)
  at monix.execution.internal.Trampoline.startLoop(Trampoline.scala:32)
  at monix.execution.schedulers.TrampolineExecutionContext$JVMOptimalTrampoline.startLoop(TrampolineExecutionContext.scala:132)
  at monix.execution.internal.Trampoline.execute(Trampoline.scala:40)
  at monix.execution.schedulers.TrampolineExecutionContext.execute(TrampolineExecutionContext.scala:57)
  at monix.execution.schedulers.BatchingScheduler.execute(BatchingScheduler.scala:50)
  at monix.execution.schedulers.BatchingScheduler.execute$(BatchingScheduler.scala:47)
  at monix.execution.schedulers.AsyncScheduler.execute(AsyncScheduler.scala:31)
  at monix.execution.schedulers.ExecuteExtensions.executeTrampolined(ExecuteExtensions.scala:86)
  at monix.execution.schedulers.ExecuteExtensions.executeTrampolined$(ExecuteExtensions.scala:85)
  at monix.execution.Scheduler$Extensions.executeTrampolined(Scheduler.scala:305)
  at monix.reactive.observables.ChainedObservable$.subscribe(ChainedObservable.scala:69)
  at monix.reactive.internal.operators.ConcatObservable.unsafeSubscribeFn(ConcatObservable.scala:37)
  at monix.reactive.observables.ChainedObservable.unsafeSubscribeFn(ChainedObservable.scala:55)
  at monix.reactive.internal.operators.OnErrorRecoverWithObservable$$anon$1.$anonfun$onError$1(OnErrorRecoverWithObservable.scala:57)
  at monix.execution.Ack$$anon$1.run(Ack.scala:54)
  at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
  at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
  at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
  at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
  at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
  at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146)
  at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
  at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
  ... 42 more
Caused by: java.io.IOException: Too many open files
  at sun.nio.ch.IOUtil.makePipe(Native Method)
  at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
  at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
  at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
  ... 46 more
any advice / direction is appreciated
Wojtek Pituła
@Krever
hey, Im runnign sttp 1.x alongside sttp 2.x, both with prometheus backend and the clash with Collector already registered that provides name: sttp_request_latency_count. I know I can override those names, but maybe there is some other way?
1 reply
Glen Marchesani
@fizzy33
So for my issue I added this (a memoize on the backend task which makes sense (I only need one backend AFAICT) --
val instance: Task[sttp.client.SttpBackend[Task, monix.reactive.Observable[java.nio.ByteBuffer], WebSocketHandler]] =
    AsyncHttpClientMonixBackend().memoize
1 reply
Glen Marchesani
@fizzy33
yeah that fixed it for me
Glen Marchesani
@fizzy33
anyone recommend code for taking a
    val webSocketT: Task[WebSocket[Task]] = ...
12 replies
and getting a
      lazy val frames: Observable[WebSocketFrame.Incoming] =