Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 12:02
    pquentin synchronize #139
  • 11:22
    pquentin opened #139
  • 11:20
    pquentin opened #138
  • 10:12
    pquentin edited #114
  • 10:08
    decentral1se synchronize #1310
  • 10:06
    decentral1se synchronize #1310
  • 07:37

    dependabot-preview[bot] on pip

    (compare)

  • 07:37
    dependabot-preview[bot] closed #54
  • 07:37
    dependabot-preview[bot] labeled #66
  • 07:37
    dependabot-preview[bot] opened #66
  • 07:37

    dependabot-preview[bot] on pip

    Bump pytest from 5.2.1 to 5.2.4… (compare)

  • 06:34
    dependabot-preview[bot] synchronize #1306
  • 06:34

    dependabot-preview[bot] on pip

    Bump pylint from 2.4.2 to 2.4.4… (compare)

  • 06:34
    dependabot-preview[bot] edited #1306
  • 06:32
    dependabot-preview[bot] edited #1306
  • 06:32

    dependabot-preview[bot] on pip

    (compare)

  • 06:32

    dependabot-preview[bot] on master

    Bump pyopenssl from 19.0.0 to 1… Merge pull request #1312 from p… (compare)

  • 06:32
    dependabot-preview[bot] closed #1312
  • 06:29

    pquentin on bleach-spike

    Add missing async/await annotat… Add sample trio test Acknowledge Python 3.6+ support… and 1 more (compare)

  • 06:29
    pquentin closed #122
Tim Stumbaugh
@tjstum_gitlab
The usual way to handle this sort of situation is to have a parser, or something that's keeping track of how much data (how many bytes) there "should be"
and in your case, not calling send_jumbo until you have determined that you have a complete message. One very customary approach is to send a 4-byte length prefix, followed by the bytes. That way, the receiver can figure out where one "message" ends.
So I guess this is all a very long-winded way of saying "you need to add code to decipher the stream length and buffer"
Will Clark
@willcl-ark
Thanks @tjstum_gitlab that's very helpful. So are you saying that the async for data in server_stream: on L37 is waiting until the stream is ending on it's own (as you have in your pseudocode), because that seems to conflict with the second part of your response :)
Tim Stumbaugh
@tjstum_gitlab
Each time you go through the for loop, you get "some amount of the data." If the stream has completely "finished" (usually because your peer called .close or the network broke or something), then the for loop terminates
So yes, I think...
Will Clark
@willcl-ark
OK understood, so I need to parse the length of the stream, buffer it, and receive until I have the correct amount before sending, inside of that for loop
Tim Stumbaugh
@tjstum_gitlab
L45 seems to be immediately sending whatever recently arrived chunk you've gotten (after preceding it with the MAGIC value, of course)
Will Clark
@willcl-ark
yep, that's where I've gone wrong then
Do you happen to know how most people get the length of a regular TCP stream? is there something in trio or another library, or do I need to be struct.unpack-ing it manually?
Tim Stumbaugh
@tjstum_gitlab
python-trio/trio#796 is the relevant issue in trio here
I've used approaches ranging from a stateful parser ("sans I/O" is a relevant term here) to just a dumb function that tries to decode an entire bytestring in one shot (and returns None if it's incomplete)
the latter is simple to implement but accidentally quadratic
NoskLo
@nosklo_gitlab
There is no way to get the length of a TCP stream. It has no predefined length. It could go on forever.
That's why you need a higher level protocol on top to do the framing
Nathaniel J. Smith
@njsmith
I guess if you only want to transmit one message, you could define your protocol so that it just sends all the data and then closes the connection, and the receiving side would handle it by just reading repeatedly until the connection is reported closed
That's how early versions of HTTP worked
But it's not a very popular approach, because usually you want to send multiple messages and you don't want to have to set up a new connection for each one
And because the recipient might not be able to tell the difference between a successful transmission, versus one where they only received half the message and then the connection broke due to an error
John Belmonte
@belm0
for busy folks I've indexed the interesting bits of my trio talk
https://trio.discourse.group/t/talk-productive-concurrency-with-trio-pycon-jp/210/2
Justin Turner Arthur
@JustinTArthur
Thanks for sharing
Nathaniel J. Smith
@njsmith
@belm0 do you have a twitter handle I should include when I tweet about it? :-)
@belm0 btw next time you need a copy of the trio logo, there are vector versions in the repo
Justin Turner Arthur
@JustinTArthur
I’ve been considering SCTP support in the stdlib (socket module up through asyncio). Is there any precedent for a langauge runner having an implementation for an IP protocol the OS/clib is missing? e.g. if on Windows or macOS, IPPROTO_SCTP isn’t available, the stdlib could support socket creation with IPPROTO_SCTP, but behind the scenes the OS call is for SOCK_RAW/IPPROTO_RAW and the socket module actually provides the SCTP implementation. Does that sound terrible? Would there be precedent for something like Trio to supply that implementation if the OS and stdlib don’t?
Nathaniel J. Smith
@njsmith
@JustinTArthur the socket module is a pretty direct exposure of the underlying system's socket API, so I don't think there's precedent for something like that at that level. Also I'm guessing the cpython team might be reluctant to take responsibility for shipping a SCTP implementation inside python.
Justin Turner Arthur
@JustinTArthur
I’m getting that impression too. How would you feel about Trio or asyncio shipping with one?
Nathaniel J. Smith
@njsmith
asyncio is part of cpython, so it has the same issues. But you could make a SCTP-on-asyncio library and put it on PyPI.
For Trio we have a bit more flexibility; we could either support it directly in the trio package, or as a third-party library sort of like trio-websocket. It's more of a judgement call on what we think will be best for users.
my first thought is that a separate library might actually be better, since SCTP is a somewhat specialized thing, so that way it gets more TLC from the folks who are experts in SCTP?
JustinTArthur @JustinTArthur nods
Nathaniel J. Smith
@njsmith
but I don't actually know that much about SCTP, so that's just a vague guess :-)
definitely the trio package should give you everything you need to implement SCTP-on-trio, and if it doesn't then we should fix that
Justin Turner Arthur
@JustinTArthur
On a similar vein, do you anticipate Trio having higher-level APIs for UDP or Unix Datagram at some point?
Nathaniel J. Smith
@njsmith
the only reason it doesn't is because I don't know what that would look like :-)
maybe the trio.socket API is already what UDP users want? UDP is a pretty specific and low-level thing (compared to say TCP, which is just one example out of many streaming byte transports)
so the answer is, it just depends on whether someone comes up with a compelling higher-level API
Justin Turner Arthur
@JustinTArthur
trio.socket may truly be enough. The only thing I can think of is that the standard socket API might seem intimidating to someone who doesn’t know how to interact with it. Looking at asyncio’s datagram support, it’s a pretty thin wrapper on the socket API, so maybe there’s not much that needs to be done.
Nathaniel J. Smith
@njsmith
@JustinTArthur it definitely is intimidating, but maybe UDP ought to be a little intimidating :-)
Justin Turner Arthur
@JustinTArthur
There might be opportunity for Trio magic though. For SCTP’s SOCK_SEQPACKET type, and maybe even datagrams in some cases, Trio’s SendChannel[bytes] feels like a potential fit as a high-level API primitive.
Nathaniel J. Smith
@njsmith
yeah
(Linux also has SOCK_SEQPACKET for unix-domain sockets, for in-order delivery with framing, which can be super useful if you're OK with it being linux-specific. The key feature is that each send is atomic, so you can have multiple processes acting as producers and consumers on the same socket.)
Justin Turner Arthur
@JustinTArthur
Linux has all the neat things, network-wise, it seems.
Nathaniel J. Smith
@njsmith
I guess that's the advantage of running everyone's datacenters, yeah
but yeah, SOCK_SEQPACKET is a Channel[bytes] (or whatever we end up calling it...), and if you're using a UDP socket for point-to-point communication only, you could model that as a Channel[bytes], though more generally I guess it's a Channel[Tuple[address, bytes]]
(for point-to-point UDP, I'm imagining a class where you specify who you're talking to when its instantiated, and then it automatically addresses outgoing packets to them, and automatically discards incoming packets from anyone else.)
Justin Turner Arthur
@JustinTArthur
Yes, I think if there were a higher-level API, one of the few advantages might be establishing the endpoint ahead of time to avoid such a tuple.
Nathaniel J. Smith
@njsmith
right. But then it's not just a higher-level API, it's a more specialized API, so there's a question of whether the point-to-point case happens often enough to justify it, and I don't know the answer to that
and I also don't know how often people use UDP without also wanting to do other messy low-level socket stuff like setting socket options or passing flags to recvmsg
Justin Turner Arthur
@JustinTArthur
I encounter the case a lot, but my industry may be niche in that regard.
(video conferencing)