IndySockets/Indy

@klsyzzz OpenSSL certificate validation with Indy is a little bit tricky (or maybe its the OpenSSL part that makes it tricky^^), sslvrfPeer says "if the server sends a certificate, it wil be verified. If the verification fails, the handshake will be terminated immediately. The only time that a server would not send a certificate is when an anonymous cipher is in use" (which should never be used^^). sslvrfFailIfNoPeerCert and sslvrfClientOnce are only used for server. If no VerifyMode is set, OpenSSL (in client mode) will verify the server certificate, but failure will not terminate the handshake.
I even believe you have always to implement a OnVerifyPeer, otherwise Indy uses a default implementation which is Result := True which overrides the verification result of OpenSSL.
If OpenSSL should verify the certificate, it needs to know which certificate are trustable. For this OpenSSL uses all public certs which are stored in the directory which you specified in SSLOptions.VerifyDir. If you want to use the windows certificate store, you could use this snippet, but you have to have a look into the msdn for CertOpenSystemStore, CertEnumCertificatesInStore and maybe is CertEnumSystemStore interessting for you.

    LCert := CertEnumCertificatesInStore(LStore, nil);
    while Assigned(LCert) do
    begin
      LX509 := d2i_X509(nil, @lCert.pbCertEncoded, LCert.cbCertEncoded);
      if Assigned(LX509) then
      begin
        X509_STORE_add_cert(ctx.cert_store, LX509);
        X509_free(LX509);
      end;
      LCert := CertEnumCertificatesInStore(LStore, LCert);
    end;
    // Calls of CertFreeCertificateContext are not needed, because
    // CertEnumCertificatesInStore frees the pPrevCertContext Argument

Another Point for certificate verification is the VerifyDepth: OpenSSL uses as default 9, but Indy overrides this with its own default, which is default(Integer) = 0. If the SMTP Server sends a certificate issued by a intermediate CA, OpenSSL terminate the connection with cert chain too long. VerifyDepth specifies the max length of a certificate chain.