These are chat archives for SmingHub/Sming

25th
Oct 2017
bbourdel
@bbourdel
Oct 25 2017 15:34
Hello @slaff,
I finally found the time to test your file upload system. Unfortunalty, it doesn't seem to work :-(
Here what I have, modify version from Basic_WebClient :
void connectOk(IPAddress ip, IPAddress mask, IPAddress gateway)
{

    debugf("Connected. Got IP: %s", ip.toString().c_str());

    HttpHeaders requestHeaders;
    requestHeaders["User-Agent"] = "WebClient/Sming";

    httpClient.send(
            httpClient.request("https://posttestserver.com/post.php")
            ->setHeaders(requestHeaders)
            ->setSslOptions(SSL_SERVER_VERIFY_LATER)
            ->onRequestComplete(onDownload)
    );


    FileStream* fileStream = new FileStream("data.txt");

    HttpHeaders fileHeaders;
    fileHeaders["Content-Type"]  = "application/octet-stream";

    httpClient.send(
            httpClient.request("https://posttestserver.com/post.php")
            ->setMethod(HTTP_POST)
            ->setHeaders(fileHeaders)
            ->setBody(fileStream)
            ->onRequestComplete(onDownload)
    );


    httpClient.send(
            httpClient.request("https://posttestserver.com/post.php")
            ->setMethod(HTTP_POST)
            ->onRequestComplete(onDownload)
    );

}
bbourdel
@bbourdel
Oct 25 2017 15:42
And this is my log :
Connected. Got IP: 10.X.X.X
Creating new httpConnection
HttpConnection::connect: TCP state: -1, isStarted: 0, isActive: 0
HttpConnection::connecting ...
+TCP connection
connect to: posttestserver.com
attached file: data.txt (19522 bytes)
HttpConnection::connect: TCP state: 0, isStarted: 1, isActive: 0
HttpConnection::connect: TCP state: 0, isStarted: 1, isActive: 0
DNS record found: posttestserver.com = 107.170.47.130
TcpConnection::connect result:, 0
OnConnected
staticOnConnected: useSSL: 1, Error: 0
SSL: Starting connection...
SSL: Switching to 160 MHz
SSL: handshake start (29470 ms)
SSL: Session Id Length: 0
please start sntp first !
SSL: handshake is in progress...
TCP sent: 105
please start sntp first !
TCP sent: 358
SSL: Handshake done (30487 ms).
SSL: Switching back to 80 MHz
TCP connected
HttpConnection::onReadyToSendData: waitingQueue.count: 3
Storing 50 bytes in stream
realloc 50 -> 208
Storing 29 bytes in stream
Storing 19 bytes in stream
Storing 2 bytes in stream
onReadyToSendData: 0
Written: 100, Available: 100, isFinished: 1, PushCount: 1 [TcpBuf: 2615]
TcpClient request completed
TCP sent: 165
HttpConnection::onReadyToSendData: waitingQueue.count: 2
onReadyToSendData: 2
SSL: Decrypted data len 331
The headers are complete
staticOnMessageComplete: Execution queue: 0, https://posttestserver.com:443/post.php
Got response code: 200
Success: 1
Got content starting with: Successfully dumped 0 post variables.
View it at h
CIPHER is SSL_AES128_SHA256
-----BEGIN SSL SESSION PARAMETERS-----
b95e8fd16d7047e5a9e070e44e8ce338a7a75b0cff6a9f457d1e8a2502d6c356
-----END SSL SESSION PARAMETERS-----
TCP connected
HttpConnection::onReadyToSendData: waitingQueue.count: 2
Storing 51 bytes in stream
realloc 51 -> 220
Storing 40 bytes in stream
Storing 23 bytes in stream
Storing 2 bytes in stream
onReadyToSendData: 0
Written: 116, Available: 116, isFinished: 1, PushCount: 1 [TcpBuf: 2599]
TcpClient request completed
TCP received: 331 bytes
HttpConnection::onReadyToSendData: waitingQueue.count: 1
onReadyToSendData: 1
Written: 1024, Available: 1024, isFinished: 0, PushCount: 1 [TcpBuf: 1506]
Written: 1024, Available: 1024, isFinished: 0, PushCount: 2 [TcpBuf: 413]
Written: -1, Available: 413, isFinished: 0, PushCount: 3 [TcpBuf: 413]
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Written: -1, Available: 413, isFinished: 0, PushCount: 1 [TcpBuf: 413]
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Written: -1, Available: 413, isFinished: 0, PushCount: 1 [TcpBuf: 413]
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Written: -1, Available: 413, isFinished: 0, PushCount: 1 [TcpBuf: 413]
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Written: -1, Available: 413, isFinished: 0, PushCount: 1 [TcpBuf: 413]
SSL: Got error: -3
TcpConnection::staticOnReceive: pbuf is NULL
TCP received: (null)
SSL: closing ...
done

TCP connection closing
-TCP connection
bbourdel
@bbourdel
Oct 25 2017 15:56
Any idea ? Does it work on your side ? I use https://posttestserver.com to test GET and POST request.
slaff
@slaff
Oct 25 2017 16:21
@bbourdel Why don't you test just one POST request sending the file data to the given URL?
It would be easier to check what goes wrong if something goes wrong.
slaff
@slaff
Oct 25 2017 16:32
Something like this:
    debugf("Connected. Got IP: %s", ip.toString().c_str());

    FileStream* fileStream = new FileStream("data.txt");

    HttpHeaders fileHeaders;
    fileHeaders["Content-Type"]  = "application/octet-stream";
    fileHeaders["User-Agent"] = "WebClient/Sming";

    httpClient.send(
            httpClient.request("https://posttestserver.com/post.php")
            ->setMethod(HTTP_POST)
            ->setHeaders(fileHeaders)
            ->setBody(fileStream)
            ->setSslOptions(SSL_SERVER_VERIFY_LATER)
            ->onRequestComplete(onDownload)
    );
bbourdel
@bbourdel
Oct 25 2017 16:33
You're right. I'm trying...
Connected. Got IP: 10.X.X.X
attached file: data.txt (19522 bytes)
Creating new httpConnection
HttpConnection::connect: TCP state: -1, isStarted: 0, isActive: 0
HttpConnection::connecting ...
+TCP connection
connect to: posttestserver.com
DNS record found: posttestserver.com = 107.170.47.130
TcpConnection::connect result:, 0
OnConnected
staticOnConnected: useSSL: 1, Error: 0
SSL: Starting connection...
SSL: Switching to 160 MHz
SSL: handshake start (14961 ms)
SSL: Session Id Length: 0
please start sntp first !
SSL: handshake is in progress...
TCP sent: 105
please start sntp first !
TCP sent: 358
SSL: Handshake done (15873 ms).
SSL: Switching back to 80 MHz
TCP connected
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Storing 51 bytes in stream
realloc 51 -> 220
Storing 40 bytes in stream
Storing 29 bytes in stream
Storing 23 bytes in stream
Storing 2 bytes in stream
onReadyToSendData: 0
Written: 145, Available: 145, isFinished: 1, PushCount: 1 [TcpBuf: 2567]
TcpClient request completed
TCP sent: 213
HttpConnection::onReadyToSendData: waitingQueue.count: 0
onReadyToSendData: 2
Written: 1024, Available: 1024, isFinished: 0, PushCount: 1 [TcpBuf: 1687]
Written: 1024, Available: 1024, isFinished: 0, PushCount: 2 [TcpBuf: 594]
Written: -1, Available: 594, isFinished: 0, PushCount: 3 [TcpBuf: 594]
HttpConnection::onReadyToSendData: waitingQueue.count: 0
Written: -1, Available: 594, isFinished: 0, PushCount: 1 [TcpBuf: 594]
A lot of ... HttpConnection::onReadyToSendData: waitingQueue.count: 0
Written: -1, Available: 594, isFinished: 0, PushCount: 1 [TcpBuf: 594]
HttpConnection::onReadyToSendData: waitingQueue.count: 0
Written: -1, Available: 594, isFinished: 0, PushCount: 1 [TcpBuf: 594]
TCP connection closed by timeout: 70 (from 70)
SSL: closing ...
done

TCP connection closing
-TCP connection
slaff
@slaff
Oct 25 2017 16:41
You have to check with Wireshark what is the remote server giving you back. It can be that there is a limit to the amount of post data that you can send or something else comes back. Why not write your own post.php, run it locally on your server and sniff directly the connection.

Written: -1, Available: 594, isFinished: 0, PushCount: 1

that basically means:
"Tcp, please, send 594 bytes. The TCP stacks replies with -> -1 = hm.. that will not happen right now. I need the sending queue to have enough space"

bbourdel
@bbourdel
Oct 25 2017 16:44
Agree ! But why does TCP queue never send queue ?
I try in a non-https (http) connection like this :
    httpClient.send(
            httpClient.request("http://posttestserver.com/post.php")
            ->setMethod(HTTP_POST)
            ->setHeaders(fileHeaders)
            ->setBody(fileStream)
            //->setSslOptions(SSL_SERVER_VERIFY_LATER)
            ->onRequestComplete(onDownload)
    );

onRequestComplete line is not commented.

Here the log :

connect to: posttestserver.com
DNS record found: posttestserver.com = 107.170.47.130
TcpConnection::connect result:, 0
OnConnected
staticOnConnected: useSSL: 0, Error: 0
TCP connected
HttpConnection::onReadyToSendData: waitingQueue.count: 1
Storing 51 bytes in stream
realloc 51 -> 220
Storing 40 bytes in stream
Storing 29 bytes in stream
Storing 23 bytes in stream
Storing 2 bytes in stream
onReadyToSendData: 0
Written: 145, Available: 145, isFinished: 1, PushCount: 1 [TcpBuf: 2635]
TcpClient request completed
TCP sent: 145
HttpConnection::onReadyToSendData: waitingQueue.count: 0
onReadyToSendData: 2
Written: 1024, Available: 1024, isFinished: 0, PushCount: 1 [TcpBuf: 1756]
Written: 1024, Available: 1024, isFinished: 0, PushCount: 2 [TcpBuf: 732]
Written: 732, Available: 732, isFinished: 0, PushCount: 3 [TcpBuf: 0]
TCP connection closed by timeout: 70 (from 70)
TCP connection closing
-TCP connection
bbourdel
@bbourdel
Oct 25 2017 16:50
I don't understand why we have a TcpClient request completed before the data stream transmission and why we don't have any TCP sent : XXX after this TcpClient request completed. Is it normal ?
In my opinion, the expected behaviour should be something like this :
This is not a real log file, it's, in my opinion, what should happen if it works.
onReadyToSendData: 0
Written: 145, Available: 145, isFinished: 1, PushCount: 1 [TcpBuf: 2635]
TCP sent: 145
//NOTE : Header sent, start to send Data Stream (here size(data_stream) = 2780)
Written: 1024, Available: 1024, isFinished: 0, PushCount: 1 [TcpBuf: 1756]
TCP sent: 1024
Written: 1024, Available: 1024, isFinished: 0, PushCount: 2 [TcpBuf: 732]
TCP sent: 1024
Written: 732, Available: 732, isFinished: 0, PushCount: 3 [TcpBuf: 0]
TCP sent: 732
TcpClient request completed
slaff
@slaff
Oct 25 2017 16:51
What is the size of the file that you are sending?
bbourdel
@bbourdel
Oct 25 2017 16:53
It's the data.txt in the main Gitlab repo, so 19 522 bytes
slaff
@slaff
Oct 25 2017 16:56
Ok. I can check this on a real device after too weeks, the earliest. Therefore can you please create an issue with the whole information.
bbourdel
@bbourdel
Oct 25 2017 16:56
I succeed a month ago in sending big file but using the espconn module. I use the callback espconn_sent to feed the TCP stack each time it has been sent.
Thanks @slaff. Looking forward on this subject. If I can help, feel free.
slaff
@slaff
Oct 25 2017 16:57
Do you know how to use Wireshark? If yes - Sniff the connection please
bbourdel
@bbourdel
Oct 25 2017 16:58
Ok, I'm doing it right now in non-secure http.
slaff
@slaff
Oct 25 2017 17:02
Looking at the code I can see the following:
if (stream->isFinished())
    {
        flush();
        debugf("TcpClient request completed");
        delete stream; // Free memory now!
        stream = NULL;
    }
The debug message is a bit misleading. It just means that the stream has finished. And in Http we have one stream for the headers and, if available, another stream for the content. So that message should be changed to 'TcpClient stream finished'
@bbourdel Are you using the latest develop version?
bbourdel
@bbourdel
Oct 25 2017 17:08
Yes
SDK 2.0.0
bbourdel
@bbourdel
Oct 25 2017 17:32
I will test wireshark tomorrow.