IndySockets/Indy

Indy - Internet Direct

Sergey

@icegood

@rlebeau ,
your comment makes sense in case if it hangs while calling Readable (ATimeout) in TIdIOHandler.ReadFromSource. But in reality it hangs inside call
LByteCount := ReadDataFromSource(LBuffer); a few lines below. I can imagine next multithread scenario:
1) first thread actively does IO over socket
2) second thread checks for connection and for this moment due activity of first thread Readable returns true.
3) Second thread calls ReadDataFromSource, but for this moment first thread fully has processed data => second thread hangs there.

Remobject's SuperTCPServer depends on Indy library and assumes that Indy10 supports multithread processing.

As for me fix should be in calling of just Readable inside TIdIOHandlerStack.Connected instead of ReadFromSource

Remy Lebeau

@rlebeau

@icegood It is not safe for multiple threads to read from a socket at the same time. That includes calling Connected(). For exactly the reason you mentioned - it gets the socket state out of sync. I/O must be synchronized across threads. It is safe for 1 thread to write while another thread reads, that does not need to be synchronized. But having 2+ threads reading, or 2+ threads writing, must be synchronized. On the other hand, 99% of the time there is no need to call Connected() directly, just let Indy raise an exception during normal I/O operations. That being said, having Connected() call Readable() instead of ReadFromSource() is not good enough, since a socket is marked as readable during a graceful disconnect, an actual read is needed to discover the disconnect.

Sergey

@icegood

Tnx, Remy!

Jos de Bruijn

@josdebr_twitter

@josdebr_twitter what is the EXACT error message? What does the call stack look like when the error occurs?
Hi Remy,
the exact error message is: "ERangeError raised: Range check error". I get the error after I call "IMAPClient.Retrieve(i, Msg)"
I've managed to generate a callstack, I can even get you a Eureka log file if you can read it. But here is the callstack extracted from my log:
IdGlobalProtocols.pas| |GetUniqueFileName |1644[45] |
IdGlobalProtocols.pas| |GetUniqueFileName |1599[0] |
IdGlobalProtocols.pas| |MakeTempFilename |1584[35] |
IdGlobalProtocols.pas| |MakeTempFilename |1549[0] |
IdAttachmentFile.pas |TIdAttachmentFile |PrepareTempStream |144[3] |
IdMessageClient.pas | |ProcessAttachment |840[7] |
IdMessageClient.pas | |ProcessAttachment |833[0] |
IdMessageClient.pas |TIdMessageClient |ReceiveBody |1024[115]|
IdMessageClient.pas |TIdMessageClient |ProcessMessage |1535[8] |
IdMessageClient.pas |TIdMessageClient |ProcessMessage |1527[0] |
IdMessageHelper.pas | |Internal_TIdMessageClientHelper_ProcessMessage|60[12] |
IdMessageHelper.pas | |Internal_TIdMessageClientHelper_ProcessMessage|48[0] |
IdMessageHelper.pas | |Internal_TIdMessageHelper_LoadFromStream |85[9] |
IdMessageHelper.pas | |Internal_TIdMessageHelper_LoadFromStream |76[0] |
IdMessageHelper.pas | |TIdMessageHelper_LoadFromStream |97[1] |
IdMessageHelper.pas | |TIdMessageHelper_LoadFromStream |96[0] |
IdIMAP4.pas |TIdIMAP4 |InternalRetrieve |4951[59] |
IdIMAP4.pas |TIdIMAP4 |InternalRetrieve |4892[0] |
IdIMAP4.pas |TIdIMAP4 |Retrieve |4601[2] |
IdIMAP4.pas |TIdIMAP4 |Retrieve |4599[0] |

Ryan Truran

@RyanTruran

@rlebeau I'm running into your name everywhere looking for a solution to my smtp not relaying to an address outside of my domain.

Ryan Truran

@RyanTruran

http://stackoverflow.com/questions/43742574/send-email-outside-of-domain-in-cbuilder
I have posted a question on stackoverflow with the link provided above. to summarize my issue I am trying to send an email outside of my domain. I am using TIdSMTP to connect to my smtp server. Everything works fine when I am only trying to send within my domain, but if I have to relay to another domain the email will not send. I receive a "Unable to relay" error when providing the correct username and password with an AuthType of satDefault. If I change the AuthType to satSASL and add an SASL mechanism I get an error that reads "Doesn't support AUTH or the specified SASL handlers!!"

Remy Lebeau

@rlebeau

@josdebr_twitter it is odd to see a range error in GetUniqueFileName(), the only thing I can see it doing that might cause that is it calls Ticks64() in a loop until FileExists() returns false. But I don't see Ticks64() in your call stack. And your line numbers do not coorespond to the latest version of IdGlobalProtocols.pas. What exactly is on line 1644 of your copy?

@RyanTruran Your server is refusing to relay, which could happen if you are not logging in correctly. The SASL error means there is no TIdSASL component in the TIdSMTP.SASLMechanisms property that matches what the server is expecting (see the AUTH entries in the TIdSMTP.Capabilities property after connecting and before logging in). If you use satDefault instead, try setting the TIdSMTP.ValidateAuthLoginCapability property to False, otherwise login might be skipped if TIdSMTP.Capabilities does not include an entry for the AUTH LOGIN command even if the server actually supports it (it is not a secure command, unless used over SSL/TLS, so it is not always advertised as supported)

Remy Lebeau

@rlebeau

@RyanTruran I updated my answer on SO.

Ryan Truran

@RyanTruran

@rlebeau where is the AUTH entry in the capabilities property? all I see is Fdelimiter, FLineBreak, FQuoteChar , etc. no AUTH or Fauth

Remy Lebeau

@rlebeau

@RyanTruran you need to look at the content of the list. Capabilities is a plain TStringList. It contains the strings that the SMTP server returns in reply to the EHLO command. Amongst other things, it will contain "AUTH=..." and/or "AUTH ..." strings for the supported authentication schemes

Ryan Truran

@RyanTruran

that's what I assumed. the list is empty

it should be under FDefined right?

Remy Lebeau

@rlebeau

@RyanTruran no, FDefined is only used for keeping track of which delimiter properties have been assigned values (Delimiter, StrictDelimiter, QuoteChar, NameValueSeparator, and LineBreak). The AUTH strings are in the Strings[] subproperty instead. Have you ever worked with TStringList before?

Ryan Truran

@RyanTruran

AUTH NTLM?

Remy Lebeau

@rlebeau

@RyanTruran Yes, those strings. Which means you need to add TIdSASLNTLM in the TIdSMTP.SASLMechanisms property, at least.

Ryan Truran

@RyanTruran

So will I need to create that or is there a mechanism already built that I can use.

I'm not seeing one with my version of c++ builder

I guess I need to add the source file

Remy Lebeau

@rlebeau

@RyanTruran Indy has many TIdSASL components. They should be on your IDE's Component Palette on the "Indy SASL" page. Or instantiate them in code at runtime instead. But either way, you need to built up the contents of the TIdSASLMechanisms property to point at those components, before you login to the server.

Ryan Truran

@RyanTruran

gotcha that's what I was doing before just not with NTLM as that does not show up in my Indy SASL tool pallete

Remy Lebeau

@rlebeau

@RyanTruran doesn't look like TIdSASLNTLM is registered by default. Not sure if it was ever finalized or not.

Ryan Truran

@RyanTruran

so what would I need to do to add it.

Remy Lebeau

@rlebeau

@RyanTruran unless you recompile Indy to register it, you should just add IdSASL_NTLM.pas to your project directly, then #include the resulting IdSASL_NTLM.hpp file in your code and instantiate a TIdSASLNTLM object at runtime and Add() it to the SASLMechanisms property.

Ryan Truran

@RyanTruran

so I should grab the .pas off of github right?

Remy Lebeau

@rlebeau

@RyanTruran I have no idea if TIdSASLNTLM works or not, which is probably why it is still not registered by default. Is that the only AUTH the server is reporting?

@RyanTruran Indy's code is not on GitHub. It is on AToZed's own SVN server (link is on Indy's website). GitHub is only used for chat and issue tracking

Ryan Truran

@RyanTruran

so are you with indy?

Remy Lebeau

@rlebeau

@RyanTruran yes, I am the primary developer of Indy, and one of its admins. But I'm not with AToZed. And I didn't write TIdSASLNTLM, and have no way of testing it myself (no access to any servers that use NTLM authentication).

Ryan Truran

@RyanTruran

gotcha

so when I add the .pas to the project does it generate a .hpp? that I can include?

Remy Lebeau

@rlebeau

@RyanTruran It will, when the project compiles the .pas file. C++Builder projects can include Pascal source files, and it will always compile Pascal files before C/C++ files, for exactly the reason of generating any necessary .hpp files that may be used by the C/C++ code.

Ryan Truran

@RyanTruran

which will occur when I compile the project right?

Remy Lebeau

@rlebeau

@RyanTruran yes

Jos de Bruijn

@josdebr_twitter

@rlebeau I am using 10.6.2.0.
On line 1644 i've got the following code: Result := LFName + IntToHex(LNamePart, 8) + LFQE;

Remy Lebeau

@rlebeau

@josdebr_twitter What version of Delphi are you using? What is the actual value of LNamePart when the error occurs? LNamePart is a TIdTicks, which is a UInt64. Is the value of LNamePart > High(Int64) (9223372036854775807) when the error occurs? IntToHex() has had an overload for UInt64 since XE2. Do you get the same error if you preceed that line of code with {$R-} or {$RANGECHECKS OFF}?

Justin

@klsyzzz

hi @rlebeau just wondering is it possible to enable certificate validation with ssl? We are using TidSMTP with UseTLS as ExplicitTLS and IOHandler as SocketOpenSSL, but looks like it is not validating the certificate

Remy Lebeau

@rlebeau

@klsyzzz Are you enabling the sslvrfPeer flag in the SSLIOHandler's VerifyMode, and using its OnVerifyPeer event? OpenSSL does validate certificates, but first you have to tell it to do so, and then you have a chance to look at the result of the validation and provide your own additional feedback based on your own needs/policies.

Justin

@klsyzzz

currently all options under VerifyMode is false

Remy Lebeau

@rlebeau

@klsyzzz Well, then turn them on, or at least sslvrfPeer

Justin

@klsyzzz

ok so I just need to turn VerifyPeer property, do I need to add anything to the OnVerifyPeer event?

Remy Lebeau

@rlebeau

@klsyzzz only if you don't trust OpenSSL's default validations, and/or you want to verify anything yourself.

Justin

@klsyzzz

@rlebeau thank you very much

Justin

@klsyzzz

sorry @rlebeau I'm new to this, do I need to set the property for CertFile or RootCertFile under SSLOptions? I got a SSL negotiation failed error after turn on sslverfPeer

Justin

@klsyzzz

@rlebeau the underlying error is this when debugging : error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

mezen

@mezen

@klsyzzz OpenSSL certificate validation with Indy is a little bit tricky (or maybe its the OpenSSL part that makes it tricky^^), sslvrfPeer says "if the server sends a certificate, it wil be verified. If the verification fails, the handshake will be terminated immediately. The only time that a server would not send a certificate is when an anonymous cipher is in use" (which should never be used^^). sslvrfFailIfNoPeerCert and sslvrfClientOnce are only used for server. If no VerifyMode is set, OpenSSL (in client mode) will verify the server certificate, but failure will not terminate the handshake.
I even believe you have always to implement a OnVerifyPeer, otherwise Indy uses a default implementation which is Result := True which overrides the verification result of OpenSSL.
If OpenSSL should verify the certificate, it needs to know which certificate are trustable. For this OpenSSL uses all public certs which are stored in the directory which you specified in SSLOptions.VerifyDir. If you want to use the windows certificate store, you could use this snippet, but you have to have a look into the msdn for CertOpenSystemStore, CertEnumCertificatesInStore and maybe is CertEnumSystemStore interessting for you.

    LCert := CertEnumCertificatesInStore(LStore, nil);
    while Assigned(LCert) do
    begin
      LX509 := d2i_X509(nil, @lCert.pbCertEncoded, LCert.cbCertEncoded);
      if Assigned(LX509) then
      begin
        X509_STORE_add_cert(ctx.cert_store, LX509);
        X509_free(LX509);
      end;
      LCert := CertEnumCertificatesInStore(LStore, LCert);
    end;
    // Calls of CertFreeCertificateContext are not needed, because
    // CertEnumCertificatesInStore frees the pPrevCertContext Argument

Another Point for certificate verification is the VerifyDepth: OpenSSL uses as default 9, but Indy overrides this with its own default, which is default(Integer) = 0. If the SMTP Server sends a certificate issued by a intermediate CA, OpenSSL terminate the connection with cert chain too long. VerifyDepth specifies the max length of a certificate chain.

mezen

@mezen

@rlebeau FYI: the shipped version of Indy with Delphi Berlin does not contain a IdSASL_NTLM.pas, that file is missing.

Jos de Bruijn

@josdebr_twitter

@rlebeau I am using D2005. And i've tried to add {R-} before my call to IMAPClient.retrieveMsg() but with no result. Unfortunately I cannot directly connect to this server, so I cannot debug the Indy library to see what value the different parameters are.

Jos de Bruijn

@josdebr_twitter

@rlebeau I've changed the function to use a UID for the temp file name:

    if(CreateGuid(Uid) = S_OK) then
      Result := Copy(stringreplace(GuidToString(Uid),'-','', [rfReplaceAll]), 2, 10) + LFQE;
//    Result := LFName + IntToHex(LNamePart, 8) + LFQE;
    if not FileExists(Result) then begin
      Break;
    end;

The first test of this change is hopefull, my application is again able to parse the messages with attachments.
Can you think of any problems I might run into using this change (Apart from compatibilty issues when I try to upgrade the Indy components)

Remy Lebeau

@rlebeau

@mezen the missing file is an Embarcadero issue, take it up with them. The file is in Indy's SVN repository, which they pull their releases from.

Where communities thrive

People

Repo info

Activity