Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 13:19

    dependabot-preview[bot] on pip

    (compare)

  • 13:19

    dependabot-preview[bot] on master

    Bump cffi from 1.12.3 to 1.13.0… Merge pull request #1250 from p… (compare)

  • 13:19
    dependabot-preview[bot] closed #1250
  • 13:10
    dependabot-preview[bot] labeled #1250
  • 13:10
    dependabot-preview[bot] opened #1250
  • 13:10

    dependabot-preview[bot] on pip

    Bump cffi from 1.12.3 to 1.13.0… (compare)

  • 12:29
    Coldplayer1995 synchronize #133
  • 12:26
    pquentin edited #133
  • 11:48
    Coldplayer1995 opened #133
  • 10:53
    pquentin review_request_removed #132
  • 10:53
    pquentin review_requested #132
  • 10:53
    pquentin review_requested #132
  • 10:22
    pquentin opened #132
  • 09:56
    RatanShreshtha opened #131
  • 09:51
    pquentin labeled #130
  • 09:51
    pquentin labeled #130
  • 09:51
    pquentin opened #130
  • 08:41
    RatanShreshtha edited #129
  • 07:19

    pquentin on bleach-spike

    Drop support for Python 3.4 (#1… Add changelog entry for Python … travis: bump MacPython version … and 12 more (compare)

Quentin Pradet
@pquentin
See the link above my message
Quentin Pradet
@pquentin
@bbkane also, yes, I think PyPI is using https://github.com/pypa/linehaul in production, and https://github.com/HyperionGray is using trio
@belm0 nice slides!
Nathaniel J. Smith
@njsmith
IIRC pypi is no longer using the trio-based linehaul – they used it for a few years and were really happy, but then their logging infrastructure changed and no longer needed a linehaul-shaped piece
Davide Rizzo
@sorcio
To be fair stream-of-objects reminds me more of conveyor belts than fluid flow, although I like the hydraulics metaphor of flow, pressure, etc
Will Clark
@willcl-ark
Hello. I have used Trio to make a TCP proxy, where the aim is to route the data over a mesh network and then out to the wider internet... It's all working OK, however I don't think I am handling larger streams of data properly: I can't see the receive buffer size in Trio, but I suspect that I am sending out via my external link before the entire stream is received. My example code is here: https://bpaste.net/show/em1Q . I suspect that L37-L45 is the part that is sending before the entire stream is received... So my question is, is trio automatically waiting on the full stream during for data in server_stream or is this receiving a fixed amount and do I need to add code in here to decipher the stream length and buffer it before sending onwards?
Tim Stumbaugh
@tjstum_gitlab
@willcl-ark one of the great things about trio is that there aren't really any hidden buffers. nothing is actually read from the underlying TCP connection until you call receive_some. the async for loop that the Stream interface offers is just a convenient shorthand for writing a loop sort of like this:
while True:
    data = await conn.receive_some()
    if not data:
        break
    # do something with data
TCP itself doesn't offer any assurances that one party's "send" lines up with the other part's "receive"
and there aren't any message boundaries that come "for free." so you can definitely see cases where it looks like you get "part of" one message in one call to receive_some and then the "rest of" the message in the next call (plus maybe part of the subsequent message!)
The usual way to handle this sort of situation is to have a parser, or something that's keeping track of how much data (how many bytes) there "should be"
Tim Stumbaugh
@tjstum_gitlab
and in your case, not calling send_jumbo until you have determined that you have a complete message. One very customary approach is to send a 4-byte length prefix, followed by the bytes. That way, the receiver can figure out where one "message" ends.
So I guess this is all a very long-winded way of saying "you need to add code to decipher the stream length and buffer"
Will Clark
@willcl-ark
Thanks @tjstum_gitlab that's very helpful. So are you saying that the async for data in server_stream: on L37 is waiting until the stream is ending on it's own (as you have in your pseudocode), because that seems to conflict with the second part of your response :)
Tim Stumbaugh
@tjstum_gitlab
Each time you go through the for loop, you get "some amount of the data." If the stream has completely "finished" (usually because your peer called .close or the network broke or something), then the for loop terminates
So yes, I think...
Will Clark
@willcl-ark
OK understood, so I need to parse the length of the stream, buffer it, and receive until I have the correct amount before sending, inside of that for loop
Tim Stumbaugh
@tjstum_gitlab
L45 seems to be immediately sending whatever recently arrived chunk you've gotten (after preceding it with the MAGIC value, of course)
Will Clark
@willcl-ark
yep, that's where I've gone wrong then
Do you happen to know how most people get the length of a regular TCP stream? is there something in trio or another library, or do I need to be struct.unpack-ing it manually?
Tim Stumbaugh
@tjstum_gitlab
python-trio/trio#796 is the relevant issue in trio here
I've used approaches ranging from a stateful parser ("sans I/O" is a relevant term here) to just a dumb function that tries to decode an entire bytestring in one shot (and returns None if it's incomplete)
the latter is simple to implement but accidentally quadratic
NoskLo
@nosklo_gitlab
There is no way to get the length of a TCP stream. It has no predefined length. It could go on forever.
That's why you need a higher level protocol on top to do the framing
Nathaniel J. Smith
@njsmith
I guess if you only want to transmit one message, you could define your protocol so that it just sends all the data and then closes the connection, and the receiving side would handle it by just reading repeatedly until the connection is reported closed
That's how early versions of HTTP worked
But it's not a very popular approach, because usually you want to send multiple messages and you don't want to have to set up a new connection for each one
And because the recipient might not be able to tell the difference between a successful transmission, versus one where they only received half the message and then the connection broke due to an error
John Belmonte
@belm0
for busy folks I've indexed the interesting bits of my trio talk
https://trio.discourse.group/t/talk-productive-concurrency-with-trio-pycon-jp/210/2
Justin Turner Arthur
@JustinTArthur
Thanks for sharing
Nathaniel J. Smith
@njsmith
@belm0 do you have a twitter handle I should include when I tweet about it? :-)
@belm0 btw next time you need a copy of the trio logo, there are vector versions in the repo
Justin Turner Arthur
@JustinTArthur
I’ve been considering SCTP support in the stdlib (socket module up through asyncio). Is there any precedent for a langauge runner having an implementation for an IP protocol the OS/clib is missing? e.g. if on Windows or macOS, IPPROTO_SCTP isn’t available, the stdlib could support socket creation with IPPROTO_SCTP, but behind the scenes the OS call is for SOCK_RAW/IPPROTO_RAW and the socket module actually provides the SCTP implementation. Does that sound terrible? Would there be precedent for something like Trio to supply that implementation if the OS and stdlib don’t?
Nathaniel J. Smith
@njsmith
@JustinTArthur the socket module is a pretty direct exposure of the underlying system's socket API, so I don't think there's precedent for something like that at that level. Also I'm guessing the cpython team might be reluctant to take responsibility for shipping a SCTP implementation inside python.
Justin Turner Arthur
@JustinTArthur
I’m getting that impression too. How would you feel about Trio or asyncio shipping with one?
Nathaniel J. Smith
@njsmith
asyncio is part of cpython, so it has the same issues. But you could make a SCTP-on-asyncio library and put it on PyPI.
For Trio we have a bit more flexibility; we could either support it directly in the trio package, or as a third-party library sort of like trio-websocket. It's more of a judgement call on what we think will be best for users.
my first thought is that a separate library might actually be better, since SCTP is a somewhat specialized thing, so that way it gets more TLC from the folks who are experts in SCTP?
JustinTArthur @JustinTArthur nods
Nathaniel J. Smith
@njsmith
but I don't actually know that much about SCTP, so that's just a vague guess :-)
definitely the trio package should give you everything you need to implement SCTP-on-trio, and if it doesn't then we should fix that
Justin Turner Arthur
@JustinTArthur
On a similar vein, do you anticipate Trio having higher-level APIs for UDP or Unix Datagram at some point?
Nathaniel J. Smith
@njsmith
the only reason it doesn't is because I don't know what that would look like :-)
maybe the trio.socket API is already what UDP users want? UDP is a pretty specific and low-level thing (compared to say TCP, which is just one example out of many streaming byte transports)
so the answer is, it just depends on whether someone comes up with a compelling higher-level API
Justin Turner Arthur
@JustinTArthur
trio.socket may truly be enough. The only thing I can think of is that the standard socket API might seem intimidating to someone who doesn’t know how to interact with it. Looking at asyncio’s datagram support, it’s a pretty thin wrapper on the socket API, so maybe there’s not much that needs to be done.
Nathaniel J. Smith
@njsmith
@JustinTArthur it definitely is intimidating, but maybe UDP ought to be a little intimidating :-)
Justin Turner Arthur
@JustinTArthur
There might be opportunity for Trio magic though. For SCTP’s SOCK_SEQPACKET type, and maybe even datagrams in some cases, Trio’s SendChannel[bytes] feels like a potential fit as a high-level API primitive.
Nathaniel J. Smith
@njsmith
yeah
(Linux also has SOCK_SEQPACKET for unix-domain sockets, for in-order delivery with framing, which can be super useful if you're OK with it being linux-specific. The key feature is that each send is atomic, so you can have multiple processes acting as producers and consumers on the same socket.)