Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 07:57
    pquentin opened #1339
  • 07:38

    dependabot-preview[bot] on pip

    (compare)

  • 07:38
    dependabot-preview[bot] closed #70
  • 07:38
    dependabot-preview[bot] labeled #72
  • 07:38
    dependabot-preview[bot] opened #72
  • 07:38

    dependabot-preview[bot] on pip

    Bump pytest from 5.2.1 to 5.3.2… (compare)

  • 06:56
    dependabot-preview[bot] synchronize #1306
  • 06:56

    dependabot-preview[bot] on pip

    Bump pylint from 2.4.2 to 2.4.4… (compare)

  • 06:56
    dependabot-preview[bot] edited #1306
  • 06:54
    dependabot-preview[bot] edited #1306
  • 06:54

    dependabot-preview[bot] on pip

    (compare)

  • 06:54

    dependabot-preview[bot] on master

    Bump pytest from 5.3.1 to 5.3.2… Merge pull request #1338 from p… (compare)

  • 06:54
    dependabot-preview[bot] closed #1338
  • 06:34
    dependabot-preview[bot] synchronize #1336
  • 06:34

    dependabot-preview[bot] on pip

    Bump ipython from 7.9.0 to 7.10… (compare)

  • 06:34
    dependabot-preview[bot] edited #1336
  • 06:34
    dependabot-preview[bot] synchronize #1338
  • 06:34

    dependabot-preview[bot] on pip

    Bump pytest from 5.3.1 to 5.3.2… (compare)

  • 06:34
    dependabot-preview[bot] edited #1338
  • 06:34
    dependabot-preview[bot] synchronize #1306
Will Clark
@willcl-ark
OK understood, so I need to parse the length of the stream, buffer it, and receive until I have the correct amount before sending, inside of that for loop
Tim Stumbaugh
@tjstum_gitlab
L45 seems to be immediately sending whatever recently arrived chunk you've gotten (after preceding it with the MAGIC value, of course)
Will Clark
@willcl-ark
yep, that's where I've gone wrong then
Do you happen to know how most people get the length of a regular TCP stream? is there something in trio or another library, or do I need to be struct.unpack-ing it manually?
Tim Stumbaugh
@tjstum_gitlab
python-trio/trio#796 is the relevant issue in trio here
I've used approaches ranging from a stateful parser ("sans I/O" is a relevant term here) to just a dumb function that tries to decode an entire bytestring in one shot (and returns None if it's incomplete)
the latter is simple to implement but accidentally quadratic
NoskLo
@nosklo_gitlab
There is no way to get the length of a TCP stream. It has no predefined length. It could go on forever.
That's why you need a higher level protocol on top to do the framing
Nathaniel J. Smith
@njsmith
I guess if you only want to transmit one message, you could define your protocol so that it just sends all the data and then closes the connection, and the receiving side would handle it by just reading repeatedly until the connection is reported closed
That's how early versions of HTTP worked
But it's not a very popular approach, because usually you want to send multiple messages and you don't want to have to set up a new connection for each one
And because the recipient might not be able to tell the difference between a successful transmission, versus one where they only received half the message and then the connection broke due to an error
John Belmonte
@belm0
for busy folks I've indexed the interesting bits of my trio talk
https://trio.discourse.group/t/talk-productive-concurrency-with-trio-pycon-jp/210/2
Justin Turner Arthur
@JustinTArthur
Thanks for sharing
Nathaniel J. Smith
@njsmith
@belm0 do you have a twitter handle I should include when I tweet about it? :-)
@belm0 btw next time you need a copy of the trio logo, there are vector versions in the repo
Justin Turner Arthur
@JustinTArthur
I’ve been considering SCTP support in the stdlib (socket module up through asyncio). Is there any precedent for a langauge runner having an implementation for an IP protocol the OS/clib is missing? e.g. if on Windows or macOS, IPPROTO_SCTP isn’t available, the stdlib could support socket creation with IPPROTO_SCTP, but behind the scenes the OS call is for SOCK_RAW/IPPROTO_RAW and the socket module actually provides the SCTP implementation. Does that sound terrible? Would there be precedent for something like Trio to supply that implementation if the OS and stdlib don’t?
Nathaniel J. Smith
@njsmith
@JustinTArthur the socket module is a pretty direct exposure of the underlying system's socket API, so I don't think there's precedent for something like that at that level. Also I'm guessing the cpython team might be reluctant to take responsibility for shipping a SCTP implementation inside python.
Justin Turner Arthur
@JustinTArthur
I’m getting that impression too. How would you feel about Trio or asyncio shipping with one?
Nathaniel J. Smith
@njsmith
asyncio is part of cpython, so it has the same issues. But you could make a SCTP-on-asyncio library and put it on PyPI.
For Trio we have a bit more flexibility; we could either support it directly in the trio package, or as a third-party library sort of like trio-websocket. It's more of a judgement call on what we think will be best for users.
my first thought is that a separate library might actually be better, since SCTP is a somewhat specialized thing, so that way it gets more TLC from the folks who are experts in SCTP?
JustinTArthur @JustinTArthur nods
Nathaniel J. Smith
@njsmith
but I don't actually know that much about SCTP, so that's just a vague guess :-)
definitely the trio package should give you everything you need to implement SCTP-on-trio, and if it doesn't then we should fix that
Justin Turner Arthur
@JustinTArthur
On a similar vein, do you anticipate Trio having higher-level APIs for UDP or Unix Datagram at some point?
Nathaniel J. Smith
@njsmith
the only reason it doesn't is because I don't know what that would look like :-)
maybe the trio.socket API is already what UDP users want? UDP is a pretty specific and low-level thing (compared to say TCP, which is just one example out of many streaming byte transports)
so the answer is, it just depends on whether someone comes up with a compelling higher-level API
Justin Turner Arthur
@JustinTArthur
trio.socket may truly be enough. The only thing I can think of is that the standard socket API might seem intimidating to someone who doesn’t know how to interact with it. Looking at asyncio’s datagram support, it’s a pretty thin wrapper on the socket API, so maybe there’s not much that needs to be done.
Nathaniel J. Smith
@njsmith
@JustinTArthur it definitely is intimidating, but maybe UDP ought to be a little intimidating :-)
Justin Turner Arthur
@JustinTArthur
There might be opportunity for Trio magic though. For SCTP’s SOCK_SEQPACKET type, and maybe even datagrams in some cases, Trio’s SendChannel[bytes] feels like a potential fit as a high-level API primitive.
Nathaniel J. Smith
@njsmith
yeah
(Linux also has SOCK_SEQPACKET for unix-domain sockets, for in-order delivery with framing, which can be super useful if you're OK with it being linux-specific. The key feature is that each send is atomic, so you can have multiple processes acting as producers and consumers on the same socket.)
Justin Turner Arthur
@JustinTArthur
Linux has all the neat things, network-wise, it seems.
Nathaniel J. Smith
@njsmith
I guess that's the advantage of running everyone's datacenters, yeah
but yeah, SOCK_SEQPACKET is a Channel[bytes] (or whatever we end up calling it...), and if you're using a UDP socket for point-to-point communication only, you could model that as a Channel[bytes], though more generally I guess it's a Channel[Tuple[address, bytes]]
(for point-to-point UDP, I'm imagining a class where you specify who you're talking to when its instantiated, and then it automatically addresses outgoing packets to them, and automatically discards incoming packets from anyone else.)
Justin Turner Arthur
@JustinTArthur
Yes, I think if there were a higher-level API, one of the few advantages might be establishing the endpoint ahead of time to avoid such a tuple.
Nathaniel J. Smith
@njsmith
right. But then it's not just a higher-level API, it's a more specialized API, so there's a question of whether the point-to-point case happens often enough to justify it, and I don't know the answer to that
and I also don't know how often people use UDP without also wanting to do other messy low-level socket stuff like setting socket options or passing flags to recvmsg
Justin Turner Arthur
@JustinTArthur
I encounter the case a lot, but my industry may be niche in that regard.
(video conferencing)
Nathaniel J. Smith
@njsmith
oh huh, that makes sense yeah :-)
does your networking code get to assume 1-1 communication, or does it have to be built to handle multi-party calls for generality even though they only make up a small fraction of all calls?
or hmm, I guess if you're using a hub-and-spoke topology with muxing through some central server then all the clients can assume 1-1 communication
Justin Turner Arthur
@JustinTArthur
Yep, all of the above. Hub and spoke style has to be supported in today’s modern remote/WAN times, but there are occasionally opportunities for peer-to-peer multicast within LAN or multicast at the router/multiplexer level.
Nathaniel J. Smith
@njsmith
ah, right, multicast is a whole other thing, you definitely need trio.socket for that :-)
@belm0 also fyi I showed your slides to spouse and some friends and now they're all saying things like "Squeeeee!!!!!! OMG" and looking at the different outfits