Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 21:52
    straight-shoota milestoned #13039
  • 21:52
    straight-shoota milestoned #13039
  • 21:44
    straight-shoota ready_for_review #12886
  • 21:08
    straight-shoota labeled #13040
  • 21:08
    straight-shoota labeled #13040
  • 21:08
    straight-shoota assigned #13040
  • 21:08
    straight-shoota opened #13040
  • 21:03
    straight-shoota synchronize #12886
  • 19:25
    straight-shoota closed #13030
  • 19:25
    straight-shoota closed #13032
  • 18:59
    beta-ziliani milestoned #13032
  • 18:59
    beta-ziliani milestoned #13032
  • 18:47
    straight-shoota converted_to_draft #12886
  • 18:09
    straight-shoota synchronize #12886
  • 18:08
    straight-shoota synchronize #12886
  • 16:37
    HertzDevil labeled #8422
  • 16:37
    HertzDevil labeled #8422
  • 16:21
    straight-shoota milestoned #13020
  • 16:21
    straight-shoota milestoned #13020
  • 15:00
    straight-shoota synchronize #12886
David Blurton
@davidblurton
@oprypin I see now that I should probably leave the data in a slice. I'm implementing a bittorrent client and the tracker responses have mixed utf8 and binary responses. The info hash is URL encoded hex escaped string. I've worked around the problem by going straight to the URL encoded version.
From IRC (bridge bot)
@FromIRC
<oprypin> @davidblurton, does that string actually go like \x12 ?
<oprypin> even if so, you can't rely on syntactic similarities of the language to pull you through.
David Blurton
@davidblurton
I'm not sure what you mean by that
From IRC (bridge bot)
@FromIRC
<oprypin> same.
David Blurton
@davidblurton
The \x12 would be URI escaped to give "%12" as the string you send to the tracker
so if I call URI.unescape I get back "\x12", but I can't make this string any other way...
From IRC (bridge bot)
@FromIRC
<oprypin> what use do you have from this string? none at all
David Blurton
@davidblurton
Right, which is why I said I should leave this data in a slice
But the response is mixed utf8 and binary, so it's a bit more tricky
<oprypin> no idea what the block is for, lol
<oprypin> it's for some special case handling if you need but there's a version without a block https://crystal-lang.org/api/0.23.1/URI.html#unescape%28string%3AString%2Cio%3AIO%2Cplus_to_space%3Dfalse%29-class-method
<oprypin> so if you know it's gonna be valid utf-8 you make String.new( those bytes )
David Blurton
@davidblurton
I guess I can do the same with the HTTP::Client response.body
use the version with a block to read into a slice and carefully convert to a string
Chris Watson
@watzon
Why does Array not have a filter method? Is there something similar that's just named differently?
From IRC (bridge bot)
@FromIRC
<oprypin> yeah, definitely don't keep arbitrary bytes in a string, it just happens to work but you cant rely on it because all string methods assume utf-8
<oprypin> but yeah String.new(slice_of_bytes) is the way to blindly stuff things into a string
David Blurton
@davidblurton
But the data will round trip fine if I were to convert it back to binary?
From IRC (bridge bot)
@FromIRC
<oprypin> @watzon, select
<oprypin> @davidblurton, if you put data into a string, don't modify it, and then obtain raw data from it, yes, it will probably be fine. why would you do that though?
<oprypin> by the way if you read a string from a http endpoint producing arbitrary bytes, that will NOT be fine
<oprypin> probably, i dunno. really unreliable in any case
David Blurton
@davidblurton
Right, doesn't make any sense.
I'll stick to slices
Can just read a byte at a time and see if it's utf8
From IRC (bridge bot)
@FromIRC
<oprypin> @davidblurton, ...uh no?
<oprypin> it has to be known in advance, defined by a protocol, guessing never works
David Blurton
@davidblurton
I won't be guessing. But I do need to parse the data. Some of it will be strings, some of it bytes.
From IRC (bridge bot)
@FromIRC
<oprypin> davidblurton so you have an IO and call this when you expect utf-8 encoded text https://crystal-lang.org/api/0.23.1/IO.html#read_string%28bytesize%3AInt%29%3AString-instance-method and other methods otherwise
David Blurton
@davidblurton
def get_raw_body(url) 
  HTTP::Client.get url do |response|
    length = response.headers.get("content_length")[0].to_i

    slice = Slice(UInt8).new(length)
    response.body_io.read_fully(slice)
    slice
  end
end
@oprypin How can I improve this?
From IRC (bridge bot)
@FromIRC
<oprypin> i'm at a loss, can't see an improvement
David Blurton
@davidblurton
Just leave the data in the an Array? I feel like slices are nicer though
From IRC (bridge bot)
@FromIRC
<hightower2> Hey, isn't behavior of Dir.entries( "/directory") a bit weird given that it returns "." and ".." among the results, in unclear sort order? I called it and it returned ["three", "four", "one", ".", "two", ".."]
<oprypin> but maybe you don't actually want to turn it into a slice and leave it in the IO
David Blurton
@davidblurton
That's true, I could pass the IO directly to my body parser
From IRC (bridge bot)
@FromIRC
<oprypin> hightower2, the sort order is not weird, but the .. stuff is indeed silly; unusable
<oprypin> so with the IO you can read a few bytes to a slice, then read a few bytes as a string
<oprypin> which seems to be what you want
<oprypin> but it's always weird how inconvenient it is to go from IO to slice. i guess they don't want people to be wasteful with allocations
<oprypin> like, instead of making a loop creating a new slice every time this encourages reuse
David Blurton
@davidblurton
I guess I was expecting the client response to have a raw_body property or something
From IRC (bridge bot)
@FromIRC
<oprypin> @davidblurton, but it allows streaming, it does not contain the whole response
<oprypin> abstracting streaming is the whole reason for IO to exist in addition to Slice
David Blurton
@davidblurton
right, I'm looking for a simple way to read the whole body though
I can see that IO is a nice abstraction for reading and writing, but I don't really care about streams here
From IRC (bridge bot)
@FromIRC
<oprypin> @davidblurton, but it's also a nice abstraction for protocol parsing
David Blurton
@davidblurton
I agree, I was using Char::Reader before, until I realised some of the response was binary