These are chat archives for request/request

26th
Aug 2017
Felipe Sabino
@felipesabino
Aug 26 2017 00:18
@tcurdt I suppose it is because content-length is the total number of octets, not characters.
Octets consist of eight bits and your body is a string, which javascript encoded using UTF-16, but you are probably receiving the data using UTF-8, a multi-byte encoding. In the end it is very possible that there is a mismatch of bytes needed to represent the string and the number of characters the string actually consist of.
Here are some references that would explain it much better:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length
https://stackoverflow.com/a/16271057/429521
Torsten Curdt
@tcurdt
Aug 26 2017 01:57
@felipesabino this could indeed be true if we are talking about some kind of string - but it's an image and the buffer should just be just bytes.
Felipe Sabino
@felipesabino
Aug 26 2017 12:30

@tcurdt Oh, I looked deeper and it is a weird behavior but it at least documented…

Here is the README part that is important:

encoding - encoding to be used on setEncoding of response data. If null, the body is returned as a Buffer. Anything else (including the default value of undefined) will be passed as the encoding parameter to toString() (meaning this is effectively utf8 by default). (Note: if you expect binary data, you should set encoding: null.)

So what I said earlier about strings still holds up and the problem is that request encoding is comparing strictly to null here https://github.com/request/request/blob/master/request.js#L1136 and as your params are empty, encoding is actually undefined, so request is handling your image not as a Buffer, but as an string by doing response.body.toString(self.encoding)…

So if you just send null for you encoding

request(url, {encoding: null}, function...

you will end up with a Buffer instead of a string for you body, and the length will be correct

Check this example with no encoding provided where body.length is different from content-length header: https://runkit.com/embed/51b8im0pvsuv
And this one where I explicitly set encoding: null and the values are the same: https://runkit.com/embed/0lgd6ct9ecyk

(had to use another image url for the example as yours was returning 403 to me on runkit :S)

Torsten Curdt
@tcurdt
Aug 26 2017 18:53
@felipesabino thanks for clearing this up.