@mezen
BasicAuthentication has to be set before you send a request, since it helps govern the type of authentication sent in the request. The only way to determine the server's desired authentication type(s) is to send a request, even if just HEAD, and see how the server responds. The authentication(s) are specified in the response's WWW-Authenticate and Proxy-Authenticate headers. If you plan on having TIdHTTP handle authentication, hoInProcessAuthshould always be set. At the very least, it needs to be set before the response is parsed, so you could set it dynamically in the OnHeadersAvailable event, for instance. If hoInProcessAuth is not set, you have to handle authentication manually by checking the TIdHTTP.ResponseCode and then setting the TIdHTTP.Request.Authentication or TIdHTTP.Request.CustomHeaders.Values['Authorization'] on the next request.
Or, now tried this: First GET request is as shown above. If I get a 407 back I set
Wireshark now says Indy send 4 (
hoInProcessAuth = True and BasicAuthentication = Falseand retry the GET request.Wireshark now says Indy send 4 (
MaxAuthRetries is still on default = 3) request instead of 1. Every request still has a basic proxy-authorization set.
@mezen TIdHTTP will not send a
BASIC request unless BasicAuthentication=True, or the server actually requests Basic in WWW-Authenticate or Proxy-Authenticate, or you force Request.Authentication or ProxyParams.Authentication to TIdBasicAuthentication directly. When TIdHTTP receives 407, the only ways that OnSelectProxyAuthorization would not be triggered are either 1) AuthProxyRetries has exceeded MaxAuthRetries, or 2) ProxyParams.Authentication is already assigned an auth class. After an auth class has been assigned, the only ways that OnProxyAuthorization would not be triggered are either 1) ProxyParams.ProxyPassword is blank, or 2) the authentication class doesn't request user input (TIdAuthentication.Next() does not return wnAskTheProgram). You are going to have to step into Indy's souce code with the debugger to figure out what TIdHTTP is really doing during its 407 processing. TIdHTTP should be handling the retry requests for you. Make sure the credentials you are using are accurate. If they are not working, maybe the NTLM request is being malformed, so the proxy keeps rejecting it.
I'm back on the multicast thing.. abandoned IPv6 support for now, however my test project works on Android and Windows, but not iOS
that still sucks, however
Hello! I see there's some chat about TIdHTTP and NTLM proxy handling in the chat since I last visited -- as it turns out I've run into something related this morning and have possibly a stupid question. The code in question tries to do a request via a proxy to "https://www.google.com", now it turns out that it had a slight flaw in that it was attempting to check the responsecode (expecting 200) but wasn't getting to see it because TIdHTTP.HTTPOptions did not contain hoNoProtocolErrorException, and for whatever reason the proxy server in question is now returning 407 so Indy was now throwing an exception as would be expected. Fine, so I added this into the TIdHTTP.HTTPOptions. However now the program ran into a seeming infinite loop inside the TIdHTTP code, and at first glance I don't understand how this is meant to work. Specifically in IdHTTP.pas line 2029 (Delphi 10.1 Berlin Indy), inside method "ConnectToHost" there is repeat until loop coded with "False" as the condition. Now obviously this loop will continue infinitely unless an exception is thrown, however with the code as it is currently (since adding hoNoProtocolErrorException) this does not happen and I seem to have what should (I would think) never happen -- an infinite loop in some library code? What am I doing wrong, and can anyone explain why this loop is hardcoded with "False" as the terminating condition?? Further background: I'm getting back a 407 from the proxy server, which I'm simply expecting to be able to detect in the client code by inspecting the TIdHTTP.ResponseCode. This code was supposedly working with SSPINTLM (based on comment in the code) though looking at it now I'm sort of dubious. It's possible the proxy server in question has had its config changed as well, I haven't followed this up yet either. But firstly, I'd like to understand why I'm not (at least) getting back a detectable 407 in the code (and am instead stuck in an infinite loop inside Indy?!)
The test code in question are as follows (warts and all):
var
LIdHTTP : TIdHTTP;
LSSLIOHandler : TIdSSLIOHandlerSocketOpenSSL;
LRequestStr, LResult : String;
begin
LRequestStr := 'https://www.google.com';
LIdHTTP := TIdHTTP.Create(nil);
try
// Setup of TidHTTP that supposedly works with Squid SSPINTLM authentication.
LSSLIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(LIdHTTP);
LSSLIOHandler.SSLOptions.Method := TIdSSLVersion.sslvTLSv1_2;
LIdHTTP.IOHandler := LSSLIOHandler;
LIdHTTP.HandleRedirects := True;
LIdHTTP.AllowCookies := True;
LIdHTTP.ConnectTimeout := 10000;
LIdHTTP.ReadTimeout := 10000;
LIdHTTP.Request.BasicAuthentication := True;
LIdHTTP.HTTPOptions := LIdHTTP.HTTPOptions + [hoKeepOrigProtocol] + [hoInProcessAuth] + [hoNoProtocolErrorException] + [hoWantProtocolErrorContent];
LIdHTTP.ProtocolVersion := pv1_1;
LIdHTTP.ProxyParams.ProxyServer := 'proxy';
LIdHTTP.ProxyParams.ProxyPort := 3128;
LResult := LIdHTTP.Get(LRequestStr);
CheckEquals(200, LIdHTTP.ResponseCode);
finally
LIdHTTP.Free;
end;
end;
@ByteJuggler 407 is how a proxy asks the client for authentication. That will trigger the
TIdHTTP.OnSelectProxyAuthorization event if the TIdHTTP.ProxyParams.Authentication property is not assigned, and then the TIdHTTP.OnProxyAuthorization event to ask for new credentials if needed. Neither of which you have assigned event handlers to. If authentication cannot be performed at all (unsupported auth scheme, no credentials provided, retry limit reached, etc), TIdHTTP should exit if hoNoProtocolErrorException is set, otherwise it will raise an exception. If authentication can proceed, TIdHTTP will try to perform it if hoInProcessAuth is set, otherwise it should exit and you will have to send a new request with proxy authentication added.
@ByteJuggler The loop you refer to is only run if connected to an HTTP proxy using the HTTP
CONNECT verb. And yes, it is looped, because it may take multiple HTTP requests to satisfy authentication, HTTP redirects, etc. The loop is hard-coded to "False" because the number of iterations needed is unknown. The inner body of the loop needs to decide when to exit the loop. hoNoProtocolErrorException was never intended to be used with proxy handling. The loop in question does not look at the return value of LLocalHTTP.ProcessResponse to break the loop if needed. A similar loop is in TIdCustomHTTP.DoRequest(), but it does look at the return value and act accordingly. So similar logic will have to be added to TIdCustomHTTP.ConnectToHost() when communicating with a proxy
@ByteJuggler until then, in your example, you can disable
hoNoProtocolErrorException and hoWantProtocolErrorContent, and then catch the raised EIdHTTPProtocolException. The response code will be in the exception's ErrorCode property, and any body content will be in the ErrorMessage property.
Thanks for the response, very much appreciated. I only added hoNoProtocolErrorException (and hoWantProtocolErrorContent) this morning as otherwise the code below the Get() would never run etc. and when it failed, the raised exception didn't make it particularly obvious initially what had failed where (whereas having the Check like fail would've been rather more direct) so I thought I'd rather just stop the whole exception thing and check the result code, whatever it may be.
Anyway given what you've said however I'll remove these again and make do with catching the exception instead as suggested for now.
Aside from this I'm still puzzled about the test failing now with a 407 in the first place. I'm trying to understand and reconcile your explanation with the fact that this code was working previously through this proxy, and to add which works (still) with NTLMSSP via Chrome and Firefox with no problems.
So I'm slightly puzzled regarding the auth, how this worked at all in the past then, seemingly by some fluke, since if I understand you correctly you're implying that the above code could not auth properly against an NTLM proxy due to missing event handlers/properties? Or am I misunderstanding you? (I've added IdHTTPSSPI and IdHTTPNTLM units to the above code, which was previously missing to ensure these were registered but hasn't made a difference... shouldn't the presence of these units register the necessary handlers so that hoInProcessAuth makes Indy deal with the authentication? I'm obviously being stupid about something... sorry!)
@ByteJuggler It is possible that the proxy simply wasn't asking for authentication before, and now it is. It is possible that
TIdSSPINTLMAuthentication succeeds (SSPI uses credentials stored in the calling thread, and the OS handles the actual authentication data) whereas TIdNTLMAuthentication fails (it requires username/password, and performs authentication manually). Just adding the IdAuthentication units to your uses clause to register the classes does not guarantee success, you still have to make sure things are setup correctly in TIdHTTP. There is just not enough information to go on to diagnose what the actual problem is. You are going to have to debug it further.
@rlebeau Thanks. The proxy was definitely not just allowing unauthenticated access previously. However there has been some changes to the proxy config I've been told to improve security so this adds an unknown, which apparently may result in 407 errors on some sites. Google however is not one of these.
Leaving that aside, I've however since discovered another variable, which is that Google seems to be processing the request differently to before.
That, is, it turns out that https://www.google.com now returns a redirect (HTTP 302) and not a direct HTTP 200; It looks like, with Indy, this is being somehow associated to the tail end of Authentication processing and ends up in effect appearing as a 407 failure. (As opposed to with Chrome/Firefox where it redirects and works as expected.)
If I however change the code to directly fetch the HTTP redirected URL, then Indy succeeds to fetch the page via the proxy (with NTLMSSP) without issue and the test passes.
So, there still seems to be something strange going on. All else being equal I should not arguably be seeing different behaviour between the browser and an Indy client, on the same PC, using the same Windows login, working through the same proxy that only allows NTLM authentication, I'd say, in the case of an immediate redirect, wouldn't you agree? (Again, both Indy and browser works as expected if no immediate redirect is present and you just fetch the URL directly.)
As an aside, I traced the code and Indy is detecting NTLM as the (only requested/allowable) auth protocol from the proxy and automatically opting to use TIdSSPINTLMAuthentication class, due to the registration of same in unit initialization. As you'll see I did start out adding handlers etc for OnSelectProxyAuthorization() and OnProxyAuthorization() but it turned out to be unnecessary.
Here is the modified code:
type TAuth = class
procedure DoSelectAuthorization(Sender: TObject; var AuthenticationClass: TIdAuthenticationClass; AuthInfo: TIdHeaderList);
procedure DoProxyAuthorization (Sender: TObject; Authentication: TIdAuthentication; var Handled: Boolean);
end;
procedure TAuth.DoSelectAuthorization(Sender: TObject; var AuthenticationClass: TIdAuthenticationClass; AuthInfo: TIdHeaderList);
begin
//It turns out AuthenticateClass is already set to TIdSSPINTLMAuthentication by
//TIdCustomHTTP.DoOnProxyAuthorization when this event is called.
//
//It does this by inspecting AResponse.ProxyAuthenticate
//list to see what protocols are requested/supported by the proxy and then
//trying to look up a suitable registered class to use to handle the requirement.
//
//In our case this of course contains 'NTLM' which results in TIdSSPINTLMAuthentication
//being looked up as appropriate class to use.
//TIdSSPINTLMAuthentication gets looked up because this class was previously
//automatically registered by idAuthenticationSSPI unit initialization
//via line 1320 call to RegisterAuthenticationMethod('NTLM', TIdSSPINTLMAuthentication);
//which registers it into global "AuthList". It is registered by inclusion in uses clause
//via IdAuthenticationSSPI (or IdAllAuthentications).
//Consequently we don't have to provide anything here and I instead just
//assert here that we're using the expected auth class instead:
Assert(AuthenticationClass = TIdSSPINTLMAuthentication);
end;
procedure TAuth.DoProxyAuthorization (Sender: TObject; Authentication: TIdAuthentication; var Handled: Boolean);
begin
//This is never called? (Presumably since not applicable because with SSP (single sign-on protocol)
//the NTLM response is generated automatically.)
ShowMessage(Authentication.Authentication);
Assert(Authentication.Authentication <> '');
end;
procedure TTextMagicTests.test_SSL_proxy_access_to_google_via_NTLMSSP;
var
LIdHTTP : TIdHTTP;
LSSLIOHandler : TIdSSLIOHandlerSocketOpenSSL;
LRequestStr, LResult : String;
LAuth : TAuth;
begin
//Using 'https://www.google.com' as LRequestStr results in HTTP 302 from google
//which seems to result in what ends up looking like a 407 Indy Exception to
//the Delphi code...
//
//Why is this? It seems incorrect behaviour. Indy should just deal with the
//redirect, like a browser does faced with the same situation.
//
//Probably not the right diagnosis, but currently it seems Indy is seemingly
//conflating the 302 as the tail-end "error" of proxy auth protocol (maybe?)
//and is then surfacing the whole lot as the original 407 exception/response.
//
//However when using the redirected URL instead, the proxy auth works as expected
//and no problems occur, and the test passes. So there the SSPINTLM auth works
//correctly, aside from this specfic situation, it seems.
LRequestStr := 'https://www.google.co.uk/?gfe_rd=cr&ei=aeJQWc2JM6nHXpaklfgG';
LIdHTTP := TIdHTTP.Create(nil);
LAuth := TAuth.Create;
try
LSSLIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(LIdHTTP);
LIdHTTP.IOHandler := LSSLIOHandler;
LSSLIOHandler.SSLOptions.Method := TIdSSLVersion.sslvTLSv1_2;
//The following 2 lines are unnecesary as SSPI is used automatically due to
//registration of TIdSSPINTLMAuthentication as handler for NTLM.
//The proxy below only supports NTLM as well. They are kept in for context.
LIdHTTP.OnSelectProxyAuthorization := LAuth.DoSelectAuthorization;
LIdHTTP.OnProxyAuthorization := LAuth.DoProxyAuthorization;
LIdHTTP.HandleRedirects := True;
LIdHTTP.AllowCookies := True;
LIdHTTP.ConnectTimeout := 10000;
LIdHTTP.ReadTimeout := 10000;
LIdHTTP.HTTPOptions := LIdHTTP.HTTPOptions + [hoKeepOrigProtocol] + [hoInProcessAuth];
LIdHTTP.ProtocolVersion := pv1_1;
LIdHTTP.ProxyParams.Clear;
LIdHTTP.ProxyParams.BasicAuthentication := False;
LIdHTTP.ProxyParams.ProxyServer := 'proxy';
LIdHTTP.ProxyParams.ProxyPort := 3128; try
LResult := LIdHTTP.Get(LRequestStr);
except
on E:EIdHTTPProtocolException do
begin
Fail('HTTP Error code: ' + IntToStr(E.ErrorCode));
end;
end;
CheckEquals(200, LIdHTTP.ResponseCode);
finally
LAuth.Free;
LIdHTTP.Free;
end;
end;
@ByteJuggler HTTP redirects are not part of authentication. And you can't even get an HTTP redirect in the first place if a proxy is used and you havn't authenticated with the proxy yet. Proxies don't generally use redirects, a request to connect to a server either succeeds or fails, with or without authentication. There is no way Indy would treat 302 as a 407 failure. Look at the code in
TIdHTTPProtocol.ProcessResponse(). The very first thing it does after validating the response headers is check for a 3xx redirect and handle it, otherwise it checks for 4xx authentication and handle it, otherwise it checks for 2xx success and handles it, otherwise it fails.
I am wondering why IdTCPServer1.Active := False; is not allowing me to kill all existing connections
@rlebeau I know redirects are not part of authentication(!) -- However I'm describing what appears to be happening. The bottom line remains: The same NTLM proxy with the same request (with a 302 thrown in) works via browser and does not with Indy. This seems to imply Indy somehow being improperly set up (where? how?), or at fault. But both work if I manually avoid the 302 redirect. I'll debug some more, though for the time being I've just changed the URL to avoid the problem. Thanks for your comments.
OnExecute
try
with AContext.Connection.IOHandler do
begin
CheckForDataOnSource(10);
if not InputBufferIsEmpty then
begin
RxBufStr := InputBuffer.ExtractToString(-1, IndyTextEncoding_UTF8);
Log(RxBufStr);
AContext.Binding.SendTo(AContext.Binding.PeerIP, 7, RxBufStr, Id_IPv4);
IdBuffer.Extract()
end;
end;
finally
end;
So, why would it hang when its asked to terminate its own threads... this is silly...
Why would I even be responsible to terminate internal socket connections threads...
@elekgeek why are you using code like this? This is not a good use of
TIdBuffer.ExtractString(), especially with a text encoding involved. You are reading arbitrary bytes and assuming they constitute a complete sequence of characters, which is not guaranteed, so the bytes may not decode to a string properly. And, you shouldn't be calling Binding.SendTo() directly at all (and besides, using sendto() on a TCP/IP socket is meaningless anyway, the destination parameters are ignored so it acts the same as send()). This kind of code is a good way to bypass TIdTCPServer's ability to auto-stop its client threads. It expects an exception to be raised when the socket is closed, but you are not allowing it to raise anything. The correct thing to do would be to use TIdIOHandler.ReadBytes() instead, letting it block until new bytes arrive and raise an exception if the client disconnects or the server is being shutting down. And use TIdIOHandler.Write() instead of Binding.Send/To() directly.
@elekgeek If you are just trying to implement an echoing server with logging, I would suggest using this instead:
var
RxBufStr: string;
begin
with AContext.Connection.IOHandler do
begin
CheckForDataOnSource(10);
CheckForDisconnect; // <-- add this
if not InputBufferIsEmpty then
begin
RxBufStr := InputBufferAsString(IndyTextEncoding_8Bit); // <-- don't assume any encoding
Log(RxBufStr);
Write(RxBufStr, IndyTextEncoding_8Bit); // <-- use TIdIOHandler.Write() instead
end;
end;
end;
@elekgeek Personally, I would not use a string at all:
var
RxBuf: TIdBytes;
begin
with AContext.Connection.IOHandler do
begin
ReadBytes(RxBuf, -1);
Log(RxBuf);
Write(RxBuf);
end;
end;
@elekgeek in which case, you might consider using
TIdECHOServer instead of TIdTCPServer. For logging, you can assign a TIdLog... component (TIdLogFile, TIdLogEvent, etc) to the AContext.Connection.IOHandler.Intercept property, such as in the OnConnect event.
@ByteJuggler handling redirects should not be affecting TIdHTTP's ability to handle authentication. If anything, a redirect simply resets the
AuthRetries and ProxyAuthRetries counters, giving TIdHTTP more chances to continue authentication attempts. Please don't assume what you think is happening, debug and find out exactly what is really happening.
@elekgeek the preferred way to use
TIdTCPServeris to use blocking I/O that follows a defined protocol, raising exceptions on errors/disconnects/shutdowns, and let the server handle the exceptions. If you skip that logic, you become responsible for handling certain things manually, like shutdown. For instance, you could set a variable before setting Active=False (or, just look at Active itself, since it is toggled to false before thread shutdowns occur), and then have OnExecute look at that variable, and if set then call AContext.Connection.Disconnect, or raise your own exception, before exiting the event.
@elekgeek the code I gave works just fine for shutting down clients. So, the most likely possibility is if
Log() itself is deadlocking, such as if it tries to synchronize with the main UI thread while the main UI thread is blocked waiting for the server to finish shutdown. Do not synchronize with the thread that is shutting down the server, that is a guaranteed deadlock. Either skip the synched operation during shutdown, or use a separate worker thread to shutdown the server so syncs can still be processed.
@elekgeek As for "rubbish text", that is just because of the use of the 8bit encoding, which stores each byte as-is as a separate
char and your code is not accounting for that. You can't use UTF-8 unless you ensure complete byte sequences for multi-byte characters, but your original code was not doing that. What kind of protocol is your server implementing? If the text is line-based, for instance, then you could just use TIdIOHandler.ReadLn(), in which case using UTF-8 would be OK. But if you use ReadBytes(-1) or ExtractString(-1) or InputBufferAsString then all guarantees about the completeness of multi-byte sequences go out the window
@rlebeau this has worked for me and solved my problem:
with IdTCPServer1.Contexts.LockList do
try
for iA := Count - 1 downto 0 do
begin
Context := Items[iA];
if Context = nil then
Continue;
Context.Connection.IOHandler.WriteBufferClear;
Context.Connection.IOHandler.InputBuffer.Clear;
Context.Connection.IOHandler.Close;
if Context.Connection.Connected then
Context.Connection.Disconnect;
end;
finally
IdTCPServer1.Contexts.UnlockList;
IdTCPServer1.Active := False;
end;What do you think?
This code also suggested by you works very well:
with AContext.Connection.IOHandler do
begin
CheckForDataOnSource(10);
CheckForDisconnect; // <-- add this
if not InputBufferIsEmpty then
begin
RxBufStr := InputBufferAsString(IndyTextEncoding_8Bit); // <-- don't assume any encoding
Log(RxBufStr);
Write(RxBufStr, IndyTextEncoding_8Bit); // <-- use TIdIOHandler.Write() instead
end;
end;Now my question:
- why were you not been able to suggest such a solution, no offense intended, but for sure you have a good reason.
- How can the client check if the server went down and close its own connection?
@elekgeek you don't need any of that. Just set
Active to false, it already disconnects the clients for you, and then waits for their threads to terminate. Your issue is that your code is blocking the threads from terminating correctly. That is what you need to fix properly, not hack around it.