Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 16 04:22
    rlebeau labeled #269
  • Oct 16 04:22
    rlebeau opened #269
  • Oct 08 19:00

    Fulgan on Restructure

    Bug fix for a typo in TIdIMAP4.… (compare)

  • Oct 08 19:00

    Fulgan on master

    Bug fix for a typo in TIdIMAP4.… (compare)

  • Oct 02 21:00

    Fulgan on Restructure

    Updating TIdIMAP4's InternalSea… (compare)

  • Oct 02 21:00

    Fulgan on master

    Updating TIdIMAP4's InternalSea… (compare)

  • Sep 20 21:50

    Fulgan on master

    Embarcadero patch for race cond… (compare)

  • Sep 20 21:50

    Fulgan on Restructure

    Embarcadero patch for race cond… (compare)

  • Sep 10 18:50
    rlebeau closed #268
  • Sep 10 18:50
    rlebeau commented #268
  • Sep 10 18:50

    Fulgan on Restructure

    Fix for TIdResponseHeaderInfo.S… (compare)

  • Sep 10 18:50

    Fulgan on master

    Fix for TIdResponseHeaderInfo.S… (compare)

  • Sep 10 18:49
    rlebeau labeled #268
  • Sep 10 18:49
    rlebeau labeled #268
  • Sep 10 18:49
    rlebeau assigned #268
  • Sep 10 18:49
    rlebeau review_requested #268
  • Sep 09 13:15
    gjdoornink opened #268
  • Aug 28 21:00

    Fulgan on Restructure

    Setting TIdSSLIOHandlerSocketBa… (compare)

  • Aug 28 21:00

    Fulgan on master

    Setting TIdSSLIOHandlerSocketBa… (compare)

  • Aug 28 19:42
    rlebeau milestoned #183
Remy Lebeau
@rlebeau
@mercedwang I don't see how your ReadBytes() implementation is much better than the current TIdIOHandler.ReadBytes() implementation. If you set a non-inifinite positive ReadTimeout, an EIdReadTimeout should already be getting raised by TIdIOHandler.ReadBytes() if no data arrives in the specified time period. Otherwise it blocks indefinately waiting for data (well, Indy actually blocks indefinately, yours unblocks after 24 days). All you are really doing is creating a similar loop to the one that already exists in TIdIOHandler.ReadFromSource(), except that you are forcing the loop to wake up every 5 seconds, whereas Indy lets the OS decide when to wake up. As for the Bindings collection, how much more flexible do you want it to be? It already allow you to bind IPv4 and IPv6 sockets concurrently on the same port, if that is supported by the OS. TIdCustomTCPServer.Startup() simply adds default listening sockets if you don't specify your own bindings beforehand. If you want more control, just setup the Bindings collection yourself how you want before activating the server
mercedwang
@mercedwang
The 5-second loop is to ensure that the TCPServer will complete the Shutdown procedure in a few seconds if I set Active to False on Linux. It also fulfill the large ReadTimeout (e.g. 20 min) requirement.
mercedwang
@mercedwang
It seems that the current implementation only add IPv4 binding automatically on Linux. Because adding both v4 and v6 bindings manually is quite easy, now I think the implementation need not changing. :smile:
Remy Lebeau
@rlebeau
@mercedwang As Chad stated, closing a socket in one thread should be aborting blocked socket operations in other threads. There should be no need for such a 5-sec loop during shutdown. If the socket is waiting for bytes, the closure will abort the wait with an error, causing the reading thread to terminate (unless it doesn't handle the error correctly). As for the Bindings, read the comment above Startup(): "Linux/Unix does not allow an IPv4 socket and an IPv6 socket to listen on the same port at the same time! Windows does not have that problem..." At the time that comment was written, that behavior was tested and verified as not working. Has that changed? It certainly doesn't work on Android, which runs on top of Linux. Listening on the same port with both IPv4 and IPv6 usually requires a dual-stack socket, which Indy doesn't support yet (IndySockets/Indy#29). However, Linux has a config option (/proc/sys/net/ipv6/bindv6only) that can automatically bind an IPv4 port when binding an IPv6 port without needing separate sockets or the IPV6_V6ONLY option enabled explicitly in code.
mercedwang
@mercedwang
As you know, some Linux distributions with defects do not abort blocked socket operations, so I have to workaround this problem.
mercedwang
@mercedwang
I remember both IPv4 and v6 bindings will be added automatically on Windows when TIdTCPServer.Startup is executed, and it can accept connections of both v4 and v6 TCP clients. Doesn't it support dual stack?
mercedwang
@mercedwang
In recent Linux distributions, netstat shows that sshd binds both v4 and v6 on port 22, and Embarcadero's PAServer binds both v4 and v6 on 64211.
Remy Lebeau
@rlebeau
@mercedwang Embarcadero uses Indy in PAServer. In fact, it was Embarcadero that first alerted me to the problem of not being able to bind separate IPv4 and IPv6 sockets to the same port on Linux (when they were working on adding Linux to Delphi), leading to the current implementation in Indy to disable that logic. But like I said, Linux has an option to auto-bind an IPv4 port when binding an IPv6 port, so you might be seeing that behavior at play
mercedwang
@mercedwang
@czhower I have done a lot of tests on various newest Linux distributions. If this is a bug truly, it is too usual. It occurs in all Linux platforms I tested including but not limited to: Ubuntu 16.04, Ubuntu 17.10, Fedora 27, CentOS 1708, open SUSE 42.3, Oracle Linux 7.4
Kudzu
@czhower
From my reading it appears to be a persistent and accepted behaviour in modern Linux. There are work arounds, but they are basically breaking tons of *nix code well beyond Indy that has had to be adjusted.
mercedwang
@mercedwang
@czhower I see. Thanks
Kudzu
@czhower
Remy - maybe we need to apply some fixes to work around this non standard beahvour Linux is adopting?
Remy Lebeau
@rlebeau
@czhower you mean, when Linux doesn't abort blocked operations when the socket is closed from another thread? That might be pretty hard to work around without going to non-blocking sockets, or using internal timeout loops on blocking operations.
Kudzu
@czhower
In modern Linux distros it apears so - but there are some work around we can apply into the code. Linux appears to have gained dementia and like typical C++ devs they are now apparently calling it a feature rather than a bug that it is - the original socket specs AFAIR are very clear on this and I know other POSIX and *nix distros are.
Ill lok for the ones I read before, but it loks like not just close but shutdown ash to be called.
I cant find the thread.. but basically it seems sometime around 2006 (post Kylix..) Linux started this new behavior and its now become "accepted" although its broken according to socket specs.
I cant tell how widespread it is, but its widespread enough to be in google a lot.
Kudzu
@czhower
I think we had even under Kylix days to do a small tweak for connecting, but connecting only... I dont remember though as its been about 15 years.
"Calling shutdown() before close() seems to cause the sync Receive to return 0 bytes (i.e. EOF)."
dotnet/corefx#22564
Thats the thread I was looking for
"FYI, for sync Send the behavior is similar: without shutdown it hangs, with shutdown it returns EINVAL -- presumably because the socket is no longer valid for sending."
so probably a simple fix.
Remy Lebeau
@rlebeau
@czhower TIdSocketHandle.CloseSocket() calls TIdStack.Disconnect(), and all of the TIdStack... classes (except TIdStackDotNet) call shutdown() before close(socket)() inside of their Disconnect().
Kudzu
@czhower
hmm.. someone needs to check it on Linux then....
rkmanaz
@rkmanaz

found a bug in IdSSLOpenSSL.pas
procedure DumpCert(AOut: TStrings; AX509: PX509);

->
BIO_get_mem_data( LMem, LBufPtr);
if (LLen > 0) and Assigned(LBufPtr) then begin

BIO_get_mem_data expects pointer to pointer
fixed code;
'
var
LMem: PBIO;
LLen : TIdC_INT;
LBufPtr : Pointer;
lPBPtr : Pointer;
begin
if Assigned(X509_print) then begin
LMem := BIO_new(BIO_s_mem);
try
lPBPtr := @LBufPtr;
X509_print(LMem, AX509);
LLen := BIO_get_mem_data( LMem, lPBPtr);

  if (LLen > 0) and Assigned(LBufPtr) then begin

'

mercedwang
@mercedwang
Found a few type errors in IdSSLOpenSSLHeaders.pas:
In definitions of EVP_DecryptUpdate, EVP_DecryptFinal, EVP_DecryptFinal_ex, EVP_CipherUpdate and EVP_OpenFinal, the type of outl parameter should not be TIdC_INT, but PIdC_INT.
The next_proto_select_cb function pointer field in the definition of SSL_CTX record: Its definition should be either 'outlen : PIdAnsiChar' or 'out outlen: TIdAnsiChar' .
Remy Lebeau
@rlebeau
@rkmanaz the original DumpCert code is fine. Look at the declaration of BIO_get_mem_data() in IdSSLOpenSSLHeaders.pas: function BIO_get_mem_data(b : PBIO; out pp : Pointer) : TIdC_INT; In Delphi, out: Pointer is equivilent to void** in C. The original code was already passing a pointer-to-pointer. Your change is passing a pointer-to-pointer-to-pointer instead
Remy Lebeau
@rlebeau
@mercedwang I don't have time right now to review everything you have pointed out, I'll do it tomorrow
rkmanaz
@rkmanaz
@rlebeau point beeing, the dumpcert function does not work, aka it never prints anything, because Assigned(LBufPtr) is always false.
i checked it with d7 as well as with dx
OpenSSL 1.0.2j 26 Sep 2016(VC-WIN32) compiler: cl /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo -DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DOPENSSL_USE_APPLINK -I. -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_SSL2 -DOPENSSL_NO_KRB5 -DOPENSSL_NO_JPAKE -DOPENSSL_NO_WEAK_SSL_CIPHERS -DOPENSSL_NO_STATIC_ENGINE
the used indy lib is somewhat older and abit adjusted to make it work under linux as well
but i checkede with the indy trunk and the important parts are the same
the only way it does work is, when i have a pointer var that i point to another pointer var and which one i pass to BIO_get_mem_data
under d7 nothing happens under dx i get an EA
rkmanaz
@rkmanaz
when i adjust the BIO_get_mem_data and remove the 'out' (which in my opinion is wrong - looking at linux man page for BIO_get_mem_data) then its same for d7 and dx, nothing happens cos nothing is assigned
rkmanaz
@rkmanaz
to make sure, i tested with OpenSSL 1.0.2n 7 Dec 2017(VC-WIN32) compiler: cl /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS -DDSO_WIN32 -W3 -WX -Gs0 -GF -Gy -nologo -DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -D_WINSOCK_DEPRECATED_NO_WARNINGS -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DOPENSSL_USE_APPLINK -I. -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_SSL2 -DOPENSSL_NO_KRB5 -DOPENSSL_NO_JPAKE -DOPENSSL_NO_WEAK_SSL_CIPHERS -DOPENSSL_NO_STATIC_ENGINE
same thing
Remy Lebeau
@rlebeau
@rkmanaz I found the problem, and it is not in DumpCert() at all, its original code is fine. The real problem is in BIO_get_mem_data() itself in IdSSLOpenSSLHeaders.pas. When it calls BIO_ctrl(), it is not passing the correct output pointer - Result := BIO_ctrl(b,BIO_CTRL_INFO,0,pp); should be Result := BIO_ctrl(b,BIO_CTRL_INFO,0,@pp); Looks like other functions, like BIO_get_mem_ptr and BIO_get_ssl, are also affected by a similar bug
Remy Lebeau
@rlebeau
@mercedwang I have checked in an update for the declarations you mentioned, and several others
mercedwang
@mercedwang
@rlebeau Great!
Jeroen Wiert Pluimers
@jpluimers
Remy Lebeau
@rlebeau
@jpluimers I don't visit Google groups very often
Remy Lebeau
@rlebeau
@jpluimers I'll review it when I have some time
@jpluimers I updated IndySockets/Indy#49 with the link to Paul's implementation so I don't lose it
Kudzu
@czhower
Looks cool.. definitely to look at integrating...
Remy Lebeau
@rlebeau
Indy has an SSPI implementation that it uses for TIdSSPINTLMAuthentication for NTLM over HTTP, but last time I looked at it, I seem to recall that it is not generalized enough for reuse with SChannel, which also uses SSPI. If we can finish fleshing out the IdSSPI unit with some missing pieces, and maybe port some of the code from the IdAuthenticationSSPI unit into IdSSPI or another common unit, that would go a long way to making an SChannel IOHandler easier to implement within Indy. I'm sure there are pieces of Paul's implementation that duplicate pieces that Indy already has and should reuse. I was already working on learning the SSPI API for SChannel use, but that effort is still in test code and not in Indy yet.