These are chat archives for IndySockets/Indy

23rd
Jun 2017
Walter Prins
@ByteJuggler
Jun 23 2017 10:51
Hello! I see there's some chat about TIdHTTP and NTLM proxy handling in the chat since I last visited -- as it turns out I've run into something related this morning and have possibly a stupid question. The code in question tries to do a request via a proxy to "https://www.google.com", now it turns out that it had a slight flaw in that it was attempting to check the responsecode (expecting 200) but wasn't getting to see it because TIdHTTP.HTTPOptions did not contain hoNoProtocolErrorException, and for whatever reason the proxy server in question is now returning 407 so Indy was now throwing an exception as would be expected. Fine, so I added this into the TIdHTTP.HTTPOptions. However now the program ran into a seeming infinite loop inside the TIdHTTP code, and at first glance I don't understand how this is meant to work. Specifically in IdHTTP.pas line 2029 (Delphi 10.1 Berlin Indy), inside method "ConnectToHost" there is repeat until loop coded with "False" as the condition. Now obviously this loop will continue infinitely unless an exception is thrown, however with the code as it is currently (since adding hoNoProtocolErrorException) this does not happen and I seem to have what should (I would think) never happen -- an infinite loop in some library code? What am I doing wrong, and can anyone explain why this loop is hardcoded with "False" as the terminating condition?? Further background: I'm getting back a 407 from the proxy server, which I'm simply expecting to be able to detect in the client code by inspecting the TIdHTTP.ResponseCode. This code was supposedly working with SSPINTLM (based on comment in the code) though looking at it now I'm sort of dubious. It's possible the proxy server in question has had its config changed as well, I haven't followed this up yet either. But firstly, I'd like to understand why I'm not (at least) getting back a detectable 407 in the code (and am instead stuck in an infinite loop inside Indy?!)
The test code in question are as follows (warts and all):
var
  LIdHTTP : TIdHTTP;
  LSSLIOHandler : TIdSSLIOHandlerSocketOpenSSL;
  LRequestStr, LResult : String;
begin
  LRequestStr := 'https://www.google.com';
  LIdHTTP := TIdHTTP.Create(nil);

  try
    // Setup of TidHTTP that supposedly works with Squid SSPINTLM authentication.
    LSSLIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(LIdHTTP);
    LSSLIOHandler.SSLOptions.Method := TIdSSLVersion.sslvTLSv1_2;
    LIdHTTP.IOHandler := LSSLIOHandler;

    LIdHTTP.HandleRedirects := True;
    LIdHTTP.AllowCookies := True;
    LIdHTTP.ConnectTimeout := 10000;
    LIdHTTP.ReadTimeout := 10000;
    LIdHTTP.Request.BasicAuthentication := True;
    LIdHTTP.HTTPOptions := LIdHTTP.HTTPOptions + [hoKeepOrigProtocol] + [hoInProcessAuth] + [hoNoProtocolErrorException] + [hoWantProtocolErrorContent];
    LIdHTTP.ProtocolVersion := pv1_1;
    LIdHTTP.ProxyParams.ProxyServer := 'proxy';
    LIdHTTP.ProxyParams.ProxyPort := 3128;

    LResult := LIdHTTP.Get(LRequestStr);

    CheckEquals(200, LIdHTTP.ResponseCode);
  finally
    LIdHTTP.Free;
  end;
end;
Remy Lebeau
@rlebeau
Jun 23 2017 17:16
@ByteJuggler 407 is how a proxy asks the client for authentication. That will trigger the TIdHTTP.OnSelectProxyAuthorization event if the TIdHTTP.ProxyParams.Authentication property is not assigned, and then the TIdHTTP.OnProxyAuthorization event to ask for new credentials if needed. Neither of which you have assigned event handlers to. If authentication cannot be performed at all (unsupported auth scheme, no credentials provided, retry limit reached, etc), TIdHTTP should exit if hoNoProtocolErrorException is set, otherwise it will raise an exception. If authentication can proceed, TIdHTTP will try to perform it if hoInProcessAuth is set, otherwise it should exit and you will have to send a new request with proxy authentication added.
@ByteJuggler The loop you refer to is only run if connected to an HTTP proxy using the HTTP CONNECT verb. And yes, it is looped, because it may take multiple HTTP requests to satisfy authentication, HTTP redirects, etc. The loop is hard-coded to "False" because the number of iterations needed is unknown. The inner body of the loop needs to decide when to exit the loop. hoNoProtocolErrorException was never intended to be used with proxy handling. The loop in question does not look at the return value of LLocalHTTP.ProcessResponse to break the loop if needed. A similar loop is in TIdCustomHTTP.DoRequest(), but it does look at the return value and act accordingly. So similar logic will have to be added to TIdCustomHTTP.ConnectToHost() when communicating with a proxy
Remy Lebeau
@rlebeau
Jun 23 2017 17:26
@ByteJuggler until then, in your example, you can disable hoNoProtocolErrorException and hoWantProtocolErrorContent, and then catch the raised EIdHTTPProtocolException. The response code will be in the exception's ErrorCode property, and any body content will be in the ErrorMessage property.
Remy Lebeau
@rlebeau
Jun 23 2017 17:37
@ByteJuggler I opened a new ticket: IndySockets/Indy#180
Walter Prins
@ByteJuggler
Jun 23 2017 23:33

Thanks for the response, very much appreciated. I only added hoNoProtocolErrorException (and hoWantProtocolErrorContent) this morning as otherwise the code below the Get() would never run etc. and when it failed, the raised exception didn't make it particularly obvious initially what had failed where (whereas having the Check like fail would've been rather more direct) so I thought I'd rather just stop the whole exception thing and check the result code, whatever it may be.

Anyway given what you've said however I'll remove these again and make do with catching the exception instead as suggested for now.

Aside from this I'm still puzzled about the test failing now with a 407 in the first place. I'm trying to understand and reconcile your explanation with the fact that this code was working previously through this proxy, and to add which works (still) with NTLMSSP via Chrome and Firefox with no problems.

So I'm slightly puzzled regarding the auth, how this worked at all in the past then, seemingly by some fluke, since if I understand you correctly you're implying that the above code could not auth properly against an NTLM proxy due to missing event handlers/properties? Or am I misunderstanding you? (I've added IdHTTPSSPI and IdHTTPNTLM units to the above code, which was previously missing to ensure these were registered but hasn't made a difference... shouldn't the presence of these units register the necessary handlers so that hoInProcessAuth makes Indy deal with the authentication? I'm obviously being stupid about something... sorry!)