hyperium/hyper

An HTTP library for Rust

Jeffrey Smith

@jeffsmith82

that works perfectly. cheers for the help. still wrapping my head around ownership in rust.

Mayank Shah

@mayankshah1607

Good day, everyone.
I'm trying to contribute to the library by working on this issue - hyperium/hyper#2178
I understand the context of the issue, but I'm a little confused about where this AddOrigin utility must be implemented (I'm very new to Rust and this is the first time I'll be contributing to a Rust project).
I'd really appreciate if anyone could help me out a little. Thanks! :)

Mayank Shah

@mayankshah1607

Hello. Is there a better place to ask the above question? - I had a couple of questions regarding the issue. I commented on the issue thread but haven't gotten any response.

Thanks again! :)

Jasper

@jbg

@mayankshah1607 my advice would be to make a PR implementing your idea of how it should be done

there isn't a single "right answer", and until there is code to look at, the discussion is quite abstract and thus less likely to get engagement from other people involved in the project

Peter Mertz

@trezm

qq: how would I go about sending trailers on an http2 response from the server? Was trying to figure it out from the docs but can't seem to find how to add them to a Body.

Peter Mertz

@trezm

FWIW I made a wrapper around Body and implemented the HttpBody trait where I just manually track and return trailer values

Jeffrey Smith

@jeffsmith82

I'm trying to get the full body of a request using
let whole_body = hyper::body::to_bytes(req.into_body()).await?;

I then try and pass &req into another function and it says it cant as it was moved by into_body(). Can i split these into two so i could just pass the headers through to the function ?

Jeffrey Smith

@jeffsmith82

so looks like
let (parts, body) = req.into_parts();
then pass the body into
let whole_body = hyper::body::to_bytes(body).await?;
and then can pass the parts bit around.

Jasper

@jbg

yes, that's correct

it's a common idiom in rust that methods named into_*() take self by value (in other words, they move/consume what they're called on)

Frank Prößdorf

@endor

I'm trying to parse a (big) XML file, do some aggregation and then send the resulting data as JSON to another service. What I did so far is parse the XML into data structures, turn the data structures into JSON, send the JSON to the other service, all one after the other. That was using quite a bit of memory and also wasn't too fast. I thought using hyper::Body::channel could help with that. So I've been using it somewhat like this: https://gist.github.com/endor/4aab0dc39f844af634b6dbf37f9ad731.

I imagine this is probably not the most efficient way to use both hyper and tokio, but I haven't really understood the concepts in tokio yet.

First I tried to send every little bit of data I just processed with sender.send_data().await, but that seemed to be very slow even though it was using less memory. Then I tried to compile larger chunks first and send those the same way and that was faster (as in total request/processing time), but was using a bit more memory.

My question is whether I'm using hyper::Body::channel correctly in combination with tokio and whether there is some way to make this still faster. For reference, the XML I'm parsing is 147MB, the peak memory usage is about 430MB and the time it takes from start to finish is 3min, which seems quite long.

Jasper

@jbg

@endor did you do any profiling (or manual timing of bits of code) to figure out where the time is spent?

it seems to me that streaming out the JSON (which sounds like what you're doing with your use of Body::channel()) is probably the least of your worries

parsing and aggregating the XML is probably where the time is spent..?

Frank Prößdorf

@endor

I did some a while ago, but just to make sure I did one again. The XML parsing and aggregation part takes about 37s. This could likely also be improved, but it still leaves ~2.5min for the JSON sending. Or rather for the JSON building from internal data structures and sending.

I was thinking that the sending takes so long, because it makes a difference in how long the whole thing takes whether I send very small pieces or whether I first build larger JSON pieces and then send those.

Frank Prößdorf

@endor

Part of that remaining 2.5min goes also to the receiving service processing the JSON, but when checking on the memory usage, it seems like the JSON building and sending part takes at least 1.5min.

rkfox

@rkfox

It's difficult to say. It seems like you have the right approach. Are you sure there isn't a network limitation sending a large amount of data?

Frank Prößdorf

@endor

These results are from running both services on localhost, so I would assume there aren’t any network limitations. It’s hood to hear that the approach is good. I will try to measure a bit more when making the pieces of data I’m sending bigger and smaller. Does it make a difference that the queue size of the hyper channel is 1 vs something bigger? Is it to be expected that sending many smaller pieces would create a noticable overhead?

rkfox

@rkfox

I think that the chunk size could have a substantial impact on the performance, yes, but it sounds like a mysterious issue. It shouldn't make a huge difference locally..

I'm not really sure what kind of overhead there is with the channel, though.

Jordan Doyle

@w4

hey guys, i've got a protocol here which doesn't exactly abide by http standards in that the client requires that a status line is sent before the body is streamed to the server, how would i go about taking control of the sockets in hyper? or at least being able to flush a response without closing the 'req.body()'

seems like as soon as a response is returned from my service req.body() wont read anything anymore

Jordan Doyle

@w4

hmm, actually it's like 100-continue functionality.. except the client doesn't send a 100-continue :P

Vitaly Shukela

@vi

Documentation for hyper::service::Service:

"It is one of Tower's fundamental abstractions."

What is Tower? Shall it be explained somehow in the doc?

Jasper

@jbg

https://docs.rs/tower

did the IRC mirror of this room move anywhere when irc.mozilla.org shut down?

the README still links to irc.mozilla.org

Gokul

@gokulchandra

Hey all I took a stab at hyperium/hyper#2178.
I went with the opinion that the Origin header made sense only when hyper is used as a client. So the Client builder now allows a configuration for setting the origin. I'd appreciate any feedback. Thanks
Pull request: hyperium/hyper#2242

Piotr Sarnacki

@drogus

Hi, I'm working on solving a problem with Hyper and I'm not sure where to look. I need to implement Server Sent Events, but I need to handle a client disconnect as fast as possible, for example when a user closes the browser tab. I found this old issue: hyperium/hyper#707, but the APIs there are pre-futures. Is there any way to detect the connection disconnect at the moment without trying to write to the stream?

Hannes de Jager

@hannesdejager

Hi there is there a way to convert a hyper client response body to a tokio:AsyncRead?

I found this ticket but it doesn't help: hyperium/hyper#1364

Hannes de Jager

@hannesdejager

Found it. Used a similar approach than this OP: https://stackoverflow.com/questions/60964238/how-to-write-a-hyper-response-body-to-a-file/63651544#63651544 (and answered him there)

inzanez

@inzanez

What might be reason that an Upgraded cannot be downcast to TcpStream?

I'm trying to upgrade a connection so that I can proxy requests between the client and another TCP service in an internal network. On the client side, I can get the raw TcpStream no problem, but the server doesn't want to downcast. I used the upgrade example as a base, which itself works fine.

piotr-roslaniec

@piotr-roslaniec

@inzanez Hello. Have you managed to solve this issue? I'm having the same problem: https://discord.com/channels/500028886025895936/670880858630258689/818223422597103668

piotr-roslaniec

@piotr-roslaniec

@inzanez It turns out we should be downcasting to AddrStream, not TcpStream. From there you can access inner TcpStream if you want:

        let upgraded_parts = upgraded.downcast::<AddrStream>().unwrap();
        let upgraded_tcp_stream = upgraded_parts.io.into_inner();

Credit to sfackler on Discord.

inzanez

@inzanez

I think I managed to solve that, yes. But I‘d need to check out how:-) But thanks for the info!

Where communities thrive

People

Repo info

Activity