softwaremill/sttp

Felix Bjært Hargreaves

@hejfelix

12:36:52.270 [AsyncHttpClient-5-2] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.ChannelPipelineException: org.asynchttpclient.netty.handler.StreamedResponsePublisher.handlerRemoved() has thrown an exception.
    at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:640) ~[netty-transport-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:477) ~[netty-transport-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:417) ~[netty-transport-4.1.60.Final.jar:4.1.60.Final]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:94) ~[async-http-client-2.12.3.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.60.Final.jar:4.1.60.Final]
...
Caused by: java.lang.IllegalArgumentException: capacity < 0: (-480651692 < 0)
    at java.base/java.nio.Buffer.createCapacityException(Buffer.java:278) ~[na:na]
    at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:360) ~[na:na]
    at sttp.client3.asynchttpclient.SimpleSubscriber.onComplete(reactive.scala:65) ~[async-http-client-backend_3-3.3.14.jar:3.3.14]
    at sttp.client3.asynchttpclient.AsyncHttpClientBackend$$anon$2.onComplete(AsyncHttpClientBackend.scala:108) ~[async-http-client-backend_3-3.3.14.jar:3.3.14]
...

8 replies

I get this when downloading large files, 3.6 gb in this case

Looks like it's causing overflow of the int used to count the size:

https://github.com/softwaremill/sttp/blob/master/async-http-client-backend/src/main/scala/sttp/client3/asynchttpclient/reactive.scala#L24

Evgenii Kuzmichev

@ekuzmichev

Hello! I'm using sttp client with AsyncHttpClientZioBackend. Version 2.2.3.
Endpoint server which I connect to responds with huge json body (90 Mb) but response time is quite fast (few seconds).
I read the ressponse with circe deserializer via .response(asJson[Option[MyDto]]). All deserilizers are correct (checked in unit tests)
I have also the .readTimeout(readTimeout) (10 min) set on my sttp request (it is definilely longer than server response time).

The problem is that I get sttp.client.DeserializationError: exhausted input due to the fact that not all bytes have been read and
I have json with cropped end and no valid json ending and etc: e.g. {"a": "foo", "b": but in scale of huger json

While debugging I've got that it is being read via zio stream instantiated from reactive streams publisher (via zio-interop-reactivestreams lib)

override protected def publisherToBytes(p: Publisher[ByteBuffer]): Task[Array[Byte]] =
    p.toStream(bufferSize).fold(ByteBuffer.allocate(0))(concatByteBuffers).map(_.array())

As I understand when requestTimeout exhausts and reading managed to start it closes the stream and leaves that bytes what it could read in this time.
And in general case it could be inconsistent, not full body completely read.

This buffer size above is 16 (private val bufferSize = 16) and I haven't found any way how to increase it.
When I Copy-Paste the code of AsyncHttpClientZioBackend to my local code base to the same package as the original one
I set larger buffer size(1024, 2048) and the exhausted input error disappears. Client manages to read full json body in time and correcly deserialize it.

Could someone help me please to handle such a problem?
Can one set buffer size via config or smth else?
Is my way of digging is correct? Maybe there is better way to solve the problem?

1 reply

Arman Bilge

@armanbilge

Hi, would someone here mind approving the workflow on my PR? :) thanks in advance!

softwaremill/sttp-model#126

Dmytro Nikitin

@KRoLer

Hey guys, I'm trying to use sttp for websocket connection on the ScalaJS side and can't figure out how can I do it with FetchBackends.
I need just open websocket connection and update the state on each received message. Will be very happy to see some kind of guide or just advice where to start

5 replies

Roman Leshchenko

@rleshchenko

Hello guys. I'm struggling with optional params in my service.
I'm specifying the route like this:

endpoint.get
        .in(
          query[Option[Filter]]("filter")
        )

And providing such a codec:

implicit val filterCodec: PlainCodec[Option[Filter]]       =
    Codec.string.mapDecode { s =>
      val l = parse(s).extract[Map[String, String]]
      Value(Option(Filter(l.head._1, l.head._2)))
    } { m =>
      Serialization.write(m)
    }

But when I'm trying to query my server with no "filter" param, I have an error:Invalid value for: query parameter filter
Any suggestions on this?

heksesang

@heksenlied:matrix.org

[m]

You wouldn't make a PlainCodec[Option[Filter]], but PlainCodec[Filter].

At least that would make more sense, I think? Doesn't tapir handle the Option-part itself?

heksesang

@heksenlied:matrix.org

[m]

I wonder if you do define PlainCodec[Option[Filter]] you might override the built-in mechanism for handling optionals. As I assume there exists a PlainCodec[Option[A]] for any A where PlainCodec[A] exists. Which means you should define a PlainCodec[Filter] and then it plays nicely.

Roman Leshchenko

@rleshchenko

@heksenlied:matrix.org, yep, you were right. Since I've deleted Option from codecs - it's started to work

Thank you very much!

Adam Warski

@adamw

@rleshchenko for the record - yes, there is built-in handling of optionals; it didn't work as your PlainCodec[Option[Filter]] defined a codec between String <-> Option[Filter]. However, query requires a codec between List[String] <-> Option[Filter] - as query parameters can be repeated. Tapir knows how to convert a String <-> T codec into List[String] <-> Option[T], but doesn't have built in conversions for String <-> Option[T]

Yuval Itzchakov

@YuvalItzchakov

Hi, is there any way to get the AsyncHttpClientZioBackend to run on the ZIO Blocking threadpool? I see the definition is: Task[SttpBackend[Task, ZioStreams with WebSockets]] although I do see in the docs it used to be a BlockingTask

3 replies

javierg1975

@javierg1975

Hi,
Is there a way to guarantee a strict order of requests?
Short of going back to a sync backend (I'm currently using cats.effects/Armeria) or introducing some sort of artificial delay between calls, I can't think of any way to achieve this.
Thanks in advance for any feedback.

5 replies

Andrii Ilin

@ilinandrii

Hey guys!
Examples for the latest documentation seems not working.
Example code: is blank

1 reply

Kopaniev Vladyslav

@VladKopanev

Hey team, sttp 3.3.15 introduced a binary incompatibility with 3.3.14, is that a known issue? Particularly it was this change:
https://github.com/softwaremill/sttp/pull/1119/files#diff-328a76b199b03a413e3a607009ce6d0ee31047193247381451e5b1362c1c96eeR15

I spotted this because my application gave me a "Exception in thread "main" java.lang.NoSuchMethodError: sttp.client3.FollowRedirectsBackend.<init>(Lsttp/client3/SttpBackend;Lscala/collection/immutable/Set;Lscala/collection/immutable/Set;)V"

(That happened after I updated one of my dependencies which transitively pulled 3.3.16 and my current version of sttp which was 3.3.14 was evicted)

1 reply

Kopaniev Vladyslav

@VladKopanev

^^ it's strange that it happened because mima plugin was enabled after 3.3.14, and if you checkout tag 3.3.15 and try to run "mimaReportBinaryIssues" it will error

Found possible hint:
https://github.com/softwaremill/sttp/runs/3776276618?check_suite_focus=true#step:8:1221
The corresponding CI build was not checking bincompat because "mimaPreviousArtifacts" was empty for some reason

Kopaniev Vladyslav

@VladKopanev

It looks like even for releases mima won't check "core" module bincompat issues:
https://github.com/softwaremill/sttp/runs/4272599204?check_suite_focus=true#step:8:1252

9 replies

Eric Meisel

@etspaceman

Hi there. I'm having a hard time upgrading from v2 to v3 due to the requirement of send to tie R to an Effect[F]

I had a class with the following definition:

class TracedLoggingSttpBackend[F[_], G[_]: Sync: Trace, +P](
    val delegate: SttpBackend[F, P],
    liftK: F ~> G
  ) extends SttpBackend[G, P]

And I was able to pass the Request from the backend to its delegate and simply run liftK on the result of it.

Now, because Request is tied to the SttpBackend's first type param (F or G), the requests become incompatible.

This was the setup we were using to tie in Natchez's kleisli usage through Sttp's backends.

4 replies

From what I can see, I need to be able to convert between the two, like so:

def convertRequest[T, RF >: P with Effect[F], RG >: P with Effect[G]](
      request: Request[T, RG]
    ): Request[T, RF]

But I have no idea how I could accomplish that

Frej Soya

@frejsoya

Simple problem here. Adding
libraryDependencies += "com.softwaremill.sttp.client3" %% "core" % "3.3.18" does not seem to update dependencies/add sttp to the classpath. Likely simple problem of me not having used scala in 5+ years :)

1 reply

Batiste Dekimpe

@batiste-dek_gitlab

Hi there, I'm running into issues using AsyncHttpClientZioBackend. I'm initiating it like that:

lazy val httpClientLayer: ULayer[Has[SttpBackend[Task, ZioStreams with capabilities.WebSockets]]] = {
    ZLayer.fromAcquireRelease(AsyncHttpClientZioBackend())(_.close().ignore).orDie
  }

and giving it to my class using Zlayer. Unfortunately, I have requests that send me the error: IllegalStateException: Closed. I'm wondering if anyone had issues with this before. Thanks a lot in advance

1 reply

Piotr Buszka

@pbuszka

I have a strange low level exception ('IndexOutOfBoundsException') in 'backend.send(req)' which appears only on 1 out of ~100 endpoints . I'm clutching at straws with this bug and the my hipotesis is that moving from sttp 3.3.4 to 3.3.18 and upgrade of scalajs-dom to 2.0.0 is the differential "cause".

sttp 3.3.4 compiled and linked with scalajs-dom 1.2 and there was no error.
sttp 3.3.4 and scalajs-dom 2.0.0 it compiles but fails on linking. 3.3.18 and 2.0.0. it compiles and linkes but systematically fails in execution on 1 particular endpoint.

I'm using FetchMonixBackend and I see in the source code that it still uses 'import org.scalajs.dom.experimental...' which is no longer available in scalajs-dom 2.0.0

Is sttp compatible with scalajs-dom 2.0.0 ?
Any clues what can I check more ?

2 replies

Andrius Bentkus

@txdv

Hi guys, how do I force sttp Uri to encode the query fragment with %?

1 reply

Tim Pigden

@TimPigden

Hi, I'm using sttp3 from zio. I'm trying to track down a bug (in my code) whereby the client thinks it's' sending a post but nothing is happening as far as the gateway logs are concerned. One of things I've realised is that 2 different areas of the code are using their own instances of AsyncHttpClientZioBackend.layer(), in one case and AsyncHttpClientZioBackend.usingConfig(config)
in another. Is this likely to cause a problem?

4 replies

Alex Myodov

@amyodov

Hello. Using sttp.client3 with AkkaHttpBackend. I see that internally, AkkaHttpBackend uses “classic” Akka model (ActorSystem, etc). So when doing AkkaHttpBackend.usingActorSystem(actorSystem), actorSystem must be a classic actor system, not a typed one. Any hints how I can take my existing typed actor system and use it in AkkaHttpBackend.usingActorSystem?

2 replies

Andrius Bentkus

@txdv

Is it possible to leave the encoding as is with sttp.url for allowed chars? for exmaple if I have https://www.test.com/A%27B, it will convert it to https://www.test.com/A'B, but I want to preserve the %27

4 replies

Stan Sobolev

@Jacke

Hi guys. Is it possible to convert sttp.monad.MonadError[sttp.client3.Identity] to cats.MonadError[sttp.client3.Identity, Throwable] w/o using effect.IO?

7 replies

Roman Makurin

@Ravenow

hi guys, is there any sport of aws signing in sttp or a way how to do it by yourself?

Igor Tovstopyat-Nelip

@igortn

Is Java 8 the strict requirement for the latest stable? Is Java 11 possible by any chance?

2 replies

For the cats-effect backend. ^

Mikhail Kobenko

@Kobenko

is there a way to disable logging of request and response body on a specific http request in Slf4jLoggingBackend?

6 replies

ex0ns

@ex0ns

I think there is an issue with release 3.4.1 of sttp for zio1.
https://github.com/softwaremill/sttp/blob/v3.4.1/build.sbt#L845 is using zio-json 0.3 which is targeting ZIO2

It should have been 0.2.0-M3 which is the latest release of zio-json for zio 1.
I've opened a PR https://github.com/softwaremill/sttp/pull/1318/files if anyone can have a looks that would be great. Thanks

1 reply

inv

@inv:envs.net

[m]

Hi! Is there something fundamental about why sttp serializes request body eagerly when constructing request instead of when sending it via backend? In my project we implemented wrapper backend which does metrics and logging besides passing request to delegate. We want to log request body, but it often contains sensitive information. Ideally, we want to implement some typeclass, say MaskedBodySerializer and use it in our wrapper backend. But when request reaches backend, its body already serialized to String. One way I see is to use tag for passing masked serialized body. Is there better approach?

2 replies

Andrii Ilin

@ilinandrii

This message was deleted

Andrii Ilin

@ilinandrii

Hi, everyone. So I've got a very strange issue with Java 11 HttpClient.
I'm using sttp client 3 with HttpClientZioBackend and calling vk.com API.
API methods have a . in the name, for example https://api.vk.com/method/photos.getAll (query params omitted)
When I call such a URI I get an error from the API which indicates that i have a problem with an API method.
However if I print url with .show and open it in browser or print curl with .toCurl and run it in console or use synchronous HttpURLConnectionBackend or AsyncHttpClientZioBackend I get a successful response.
I've also used HttpClient directly and reproduced the same problem.
Sure I've got AsyncHttpClientZioBackend as a workaround, but wanted to use client with no extra dependencies, maybe someone knows how to overcome this problem.

1 reply

Vasyl Zhurba

@vasyl-zhurba

Hi, is possible to expose Kafka events documentation via AsyncApi+Tapir (without server logic connected)? Is there an example I could take a look at? I saw websocket examples, but I couldn't adapt it to kafka events documentation

1 reply

Ori Hirshfeld

@ori888_twitter

Hi Everyone,
I'm struggling sending a request with SSL context using HttpClientZioBackend, can someone post a sample code of how it should be done?
Documentation is missing that part.
Thank you

3 replies

Christian Bartolo Burlò

@chrisbartoloburlo

Hello, I am attempting to send requests via sttp3, however I am getting the following error: Cannot prove that Any with sttp.capabilities.Effect[[X]sttp.client3.Identity[X]] <:< Nothing.

I am sending the request as follows:

val api = DefaultApi.apply()
val backend = HttpURLConnectionBackend()
println(api.addProduct("Calculator").send(backend))

where:

def addProduct(productName: String
): Request[Either[ResponseException[String, Exception], String], Nothing] =
    basicRequest
      .method(Method.POST, uri"$baseUrl/products/${productName}")
      .contentType("application/json")
.response(asJson[String])

Am I doing something wrong? Thanks!

6 replies

ex0ns

@ex0ns

The last parameters of the request is the support for capabilities (i.e streaming), if you don't support it you should pass Any and not Nothing, otherwise the compiler will not be able to derive the correct effect (as Nothing is the bottom type)
https://sourcegraph.com/github.com/softwaremill/sttp@177bb70974fbb2bb01aaeae592416743f63cba30/-/blob/core/src/main/scala/sttp/client3/RequestT.scala?L33

  * @tparam R
  *   The backend capabilities required by the request or response description. This might be `Any` (no requirements),
  *   [[Effect]] (the backend must support the given effect type), [[Streams]] (the ability to send and receive
  *   streaming bodies) or [[sttp.capabilities.WebSockets]] (the ability to handle websocket requests).

1 reply

Christian Bartolo Burlò

@chrisbartoloburlo

Hello everyone. The following code is generated using the openapi-generator:

def addRequiresConstraintToProduct(productName: String, sourceFeature: String, requiredFeature: String
): Request[Either[Either[String, String], Unit], Any] =
    basicRequest
      .method(Method.POST, uri"$baseUrl/products/${productName}/constraints/requires")
      .contentType("multipart/form-data")
      .multipartBody(Seq(
                multipart("sourceFeature", sourceFeature)
, 
                multipart("requiredFeature", requiredFeature)

      ).flatten)
.response(asEither(asString, ignore))

And it is giving the following error:

No implicit view available from sttp.model.Part[sttp.client3.BasicRequestBody] => scala.collection.IterableOnce[B]
).flatten)
  ^

Any help would be greatly appreciated. Thanks!

8 replies

Fabrice Sznajderman

@Fabszn

Hi all

I implementing a httpclient to query an API under OAuth2

I using AccessTokenProvider to request the accessToken before to make my business request

When I ran the accessToken request.. I get the following error :

com.ocadotechnology.sttp.oauth2.common$OAuth2Exception: Client call resulted in error (200): expected json value got 'grant_...

I understand than accessProvider requiring the grant_type.. But I don't know how to provide it ... :-/

Where communities thrive

People

Repo info

Activity