These are chat archives for IndySockets/Indy

1st
Mar 2017
Remy Lebeau
@rlebeau
Mar 01 2017 00:00
@SamBirnbaum read the instructions
@SamBirnbaum but first, I suggest you compare the HTTP request that the browser sends to the HTTP request that TIdHTTP sends, see what the differences are
Sam B
@SamBirnbaum
Mar 01 2017 00:04
@rlebeau Thank you very much. I will certainly read the instructions. Quick question, how can I compare what the browser sends (Fox) vs the TIdHTTP ?
Remy Lebeau
@rlebeau
Mar 01 2017 00:05
@SamBirnbaum pretty much every modern browser has a debug tool for viewing the raw HTTP messages being exchanged. On the Indy side, since you are dealing with HTTPS, you can either use a debugging proxy like Fiddler, or simply attach one of the TIdLog... components to the TIdHTTP.Intercept property.
Sam B
@SamBirnbaum
Mar 01 2017 00:07
@rlebeau Great. Thanks. I don't know where you are located, but here its diner time. Being called down for that. I will keep you updated. Thanks again.
Sam B
@SamBirnbaum
Mar 01 2017 02:10

@rlebeau , it seems (to me anyway) that its send out the correct request. I captured the on send in the event log. I don't know what else to look for. This is a direct translation of the buffer.

GET /1.0/tops/last?symbols=AAPL,fb,AIG+
HTTP/1.1'#$D#$A'
Host: api.iextrading.com'#$D#$A'
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8'#$D#$A'
Accept-Encoding: identity'#$D#$A'
User-Agent: Mozilla/3.0 (compatible; Indy Library)'#$D#$A#$D#$A

@rlebeau Since I am using XE5, which versions should I download. The version of IDHTTP that I have is 10.6.0.5...
Remy Lebeau
@rlebeau
Mar 01 2017 02:47
@SamBirnbaum that request is fine, and is similar to what mine is sending: GET /1.0/tops/last?symbols=AAPL%2Cfb%2CAIG%2B HTTP/1.1
Host: api.iextrading.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
User-Agent: Mozilla/3.0 (compatible; Indy Library)
@SamBirnbaum Notice there is a slight difference, though. Newer versions of Indy no longer send the 'Accept-Encoding: identity' header unnecessarily, as some servers can't handle that header. Maybe your connection is passing through a proxy server that doesn't handle that header.
Sam B
@SamBirnbaum
Mar 01 2017 04:03
Ahh, is there a way, for me to strip that (in my intercept) out before the actual string is sent out ?
Kudzu
@czhower
Mar 01 2017 13:43
It might be easier to just comment it out in your source.
Sam B
@SamBirnbaum
Mar 01 2017 14:26
@czhower Hi, yes I traced it to the request structure in the HTTPheader. The problem was that the header still has the field FAcceptencoding so that the buffer being sent has the line "Accept-encoding:" without any parameter after it. That may or may not have caused the error 502 to be returned anyway. So my next quest is to try to delete that line completely (if I can). This morning before dropping of my granddaughter to school, I started to look at the source code (Indy) to see where that is being prepared so that I can comment that out completely. Thanks for the follow up response. Need all the help I can get on this one. This is not my forte, I am more of an application guy. Being doing it for 40+ years.
Sam B
@SamBirnbaum
Mar 01 2017 15:56
@czhower @rlebeau Hi, Managed to find the spot where the header is constructed so that the data looks like the one Remy had and I still received the 502 error.
'GET /1.0/tops/last?symbols=AAPL,fb,AIG+ HTTP/1.1'#$D#$A
'Host: api.iextrading.com'#$D#$A
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8'#$D#$A
'User-Agent: Mozilla/3.0 (compatible; Indy Library)'#$D#$A#$D#$A
Sam B
@SamBirnbaum
Mar 01 2017 16:18
@czhower @rlebeau Hi, I did try to use the same code without the 'Accept-Encoding' on another site and had no problems. Weird.
Sam B
@SamBirnbaum
Mar 01 2017 16:25
@czhower @rlebeau Hi, should the HTTP/1.1 appear on a separate line ?
Remy Lebeau
@rlebeau
Mar 01 2017 16:58
@SamBirnbaum no, it should not. And if you are still having problems after stripping out the 'Accept-Encoding' header then the problem lies elsewhere. I still think this is going to come down to a connection issue, not a request issue. 502 means the request is passing through a proxy/gateway that cannot forward the request to another server. That has nothing to do with TIdHTTP itself
Sam B
@SamBirnbaum
Mar 01 2017 17:22
@rlebeau Thanks. I am starting to think it might be my ISP (Optimum) but the browser has no problem getting to that site. That is what perplexing me. I captured the headers that the browser is sending and receiving. Can I send the screen shot to you or post it here, maybe you can spot something in the response that the browser is handling that is not being handled by my program due to some setting.
Remy Lebeau
@rlebeau
Mar 01 2017 18:23
@SamBirnbaum You can show the browser's request (copy/pasted text woould be better than an image, but Gitter chat does support images, too), but I don't think it is going to make much difference. The server's response is irrelevant unless the server is sending back a redirect (it wasn't when I tested it, but since you are likely passing through a proxy/gateway, you may be getting a different response than what I see).
Sam B
@SamBirnbaum
Mar 01 2017 18:34

@rlebeau Hi, here are two lines on the response that may or may not may a difference:
content-security-policy default-src 'self'; child-src 'none'; object-src 'none'; style-src 'self' 'unsafe-inline'; font-src data:; connect-src 'self' https://auth.iextrading.com wss://iextrading.com wss://tops.iextrading.com; script-src 'self';

x-content-security-policy default-src 'self'; child-src 'none'; object-src 'none'; style-src 'self' 'unsafe-inline'; font-src data:; connect-src 'self' https://auth.iextrading.com wss://iextrading.com wss://tops.iextrading.com; script-src 'self';

what I also noticed on the header going out were these two lines:

Accept-Encoding gzip, deflate, br
Upgrade-Insecure-Requests 1

How do I add an image here or can I attach a file ?

Remy Lebeau
@rlebeau
Mar 01 2017 19:21
@SamBirnHaum The 'Accept-Encoding' header is not relevant to your issue. However, if you want to enable compression, you can assign a TIdZLibCompressorBase-derived component to the TIdHTTP.Compressor property, such as TIdCompressorZLib. Indy does not support the 'br' encoding. As for the 'Upgrade-Insecure-Requests' header, you can add that using the TIdHTTP.Request.CustomHeaders property
@SamBirnhaum 'content-security-policy' is a response header, not a request header, so not relevant to your issue, either.
Sam B
@SamBirnbaum
Mar 01 2017 19:34
@rlebeau Thanks. I don't want to bother you. I try to figure out things by myself by experimenting. 99% of the time that works. I wish it would have worked this time as well. I did send you an email. Hope you got it. Thanks again.
Remy Lebeau
@rlebeau
Mar 01 2017 19:38
@SamBirnbaum I got it, and what it shows is not relevant to this issue. But if you really want to rule things out, simply make TIdHTTP send everything the browser sends, and see what happens. Like I said though, I tested it myself and it worked fine without having to do anything extra to make it work. So I still think you are looking in the wrong places for the culprit.
Sam B
@SamBirnbaum
Mar 01 2017 20:21
@rlebeau I was afraid you would might say that. This may sound silly, but can you email me a simple executable that just does the get to that site. on a button press Showmessage(get(...)); If this does not runs on my PC, then I know that its my connection.
Remy Lebeau
@rlebeau
Mar 01 2017 20:31
@SamBirnbaum done
Sam B
@SamBirnbaum
Mar 01 2017 20:32
@rlebeau Thanks, I was just about ready to send you my program to see if it works on your end.
Sam B
@SamBirnbaum
Mar 01 2017 21:06
@rlebeau Test your little program. Works like a charm.
Sam B
@SamBirnbaum
Mar 01 2017 21:19
@rlebeau I guess its time to reload Indy10 and hope it works.
@rlebeau really appreciate all the help and effort you made in this regards. Thank you very much.