These are chat archives for esp8266/Arduino

2nd
Jun 2016
Ivan Grokhotkov
@igrr
Jun 02 2016 02:58
@aneek231 we don't have gdb for Windows unfortunately...
Ivan Grokhotkov
@igrr
Jun 02 2016 03:07
@martinayotte would love to test that, unfortunately nytimes is blocked in China...
however i think there is a potential error in that sketch, while ((size = client.available()) > 0) {
for some period of time client.available() may return zero. This doesn't mean that you should bail out of the loop, if you haven't yet received all the content.
I think this should be handled correctly by HTTPClient library
Please see StreamHttpClient example
Ivan Grokhotkov
@igrr
Jun 02 2016 03:31
Okay, i changed nytimes.com to eetimes.com and got the following to work: https://gist.github.com/igrr/a81e10c37575c3ea2874d6c1a18b0eb4
Martin Ayotte
@martinayotte
Jun 02 2016 15:14
Thanks ! @igrr . But I found something really interesting : if the output is Serial, it works fine, but if the output is a Telnet client, it is slower than a snail, even after 1 minutes, the connection is still ongoing outputting less than 1 line per second.
Russ Mathis
@RussMathis
Jun 02 2016 16:02
Has anyone noticed that when a cellphone near an esp8266 the esp disconnects from its network?
sic. gets near
Ivan Grokhotkov
@igrr
Jun 02 2016 16:10
If you increase buffer size to 1460 bytes, it should get better
@RussMathis never noticed that. In fact, I have an Esp in my cellphone stand...
Poor board layout maybe?
@martinayotte less than 1 line per second? Oh...
With something like telnetClient.write(buf, size)?
Will check tomorrow
128 byte buffer should be slow, but not that slow...
Ivan Grokhotkov
@igrr
Jun 02 2016 16:16
@martinayotte could you please post the code which you were using somewhere?
Martin Ayotte
@martinayotte
Jun 02 2016 16:35
@igrr , changing the buffer to 1024 make the chunk bigger, but the delay between chunks still slow, several seconds between chunks. My Sketch-Buffet is pretty huge, but I will try to extract a tinnier version later today (with only the telnet and the http-get).
BTW, Yes, it is sent using telnetClient.write(buf, size) ...
Martin Ayotte
@martinayotte
Jun 02 2016 17:15
@igrr , here it is : https://gist.github.com/martinayotte/a399c2a4bc291dc1c25a7bf607214a17 . I've also tested connecting telnet client on AP side, the same issue is there. You will see that even the first chunk take few seconds before been printed after the "200 OK" been printed, while on Serial it is almost instantaneous.
Clemens Kirchgatterer
@everslick
Jun 02 2016 17:15
i also suspect a rather nasty bug somewhere in the stack. i see it when sending big chunks of data over websocket. at some point the browser will report a UTF8 encoding error and refuse to parse the JSON. but it is hard to make a tiny sketch to reproduce it. but it is related to buffer sizes, the bigger, the sooner it fails.
Martin Ayotte
@martinayotte
Jun 02 2016 17:17
In my case, I was able to output SPIFFS 100K files over telnet without problems, so it is maybe related to having both http client and telnet client at the same time.
Me No Dev
@me-no-dev
Jun 02 2016 18:23
and both being blocking/non-async
@martinayotte do you yield after writing to the telnet client?
http.read -> telnet.write -> yield
read/write chunks should be 1460 bytes or similar
yielding here is really important because you have to ack back to the browser what you got, send data to the telnet and at somne point get ack from the telnet that it got data. 4 packets must go through for a single packet of data to be proxied
Martin Ayotte
@martinayotte
Jun 02 2016 18:50
@me-no-dev , I've use the same piece of code that @igrr sent, except telnet.write() instead of Serial.write(), it doesn't have a yield() but a delay(1).
And, as I said, the strange thing is that there is even a first delay between the "200 OK" and the first appearing of telnet.write()
Me No Dev
@me-no-dev
Jun 02 2016 18:54
what size buffer are you using?
should be 1460
Martin Ayotte
@martinayotte
Jun 02 2016 19:13
Igrr had 128 but I've boosted to 1024. Seeing your comment, I changed it to 1460, but issue still there.
BTW, the delays are not present on every chunks, it is intermittent. The NYTimes page is about 175KB, maybe there is about 4 to 10 delays
Me No Dev
@me-no-dev
Jun 02 2016 19:18
and you say delays are around 2 seconds?
Martin Ayotte
@martinayotte
Jun 02 2016 19:19
some yes, but some other and most of them are 5 secs
Doing "wget -O - http://www.nytimes.com" on Linux is instantaneous for the whole page
It is also instantenous on ESP when sent to Serial, same code, only the Print class pointer changed
Me No Dev
@me-no-dev
Jun 02 2016 19:22
have you tried TCP dumping it?
maybe the ack does not get to the server, or is getting there late
the delays you mention are quite long
Martin Ayotte
@martinayotte
Jun 02 2016 19:27
I'm not very fluent with tcpdump.
Me No Dev
@me-no-dev
Jun 02 2016 19:28
another thing could be happening also
since you read some data from the http client and you send it to the telnet, you are waiting while the whole sending is happening.
then when new data comes, the server is probably getting ack for the previous packet and attempts to resend it
tcpdump can show what is actually happening, because I'm really guessing at this point
Martin Ayotte
@martinayotte
Jun 02 2016 19:32
I will try to get more familiar with tcpdump
(I've already used it for something else, but never in ESP context)
Me No Dev
@me-no-dev
Jun 02 2016 19:33
I can help you decode the information
are you familiar with TCP connection flow?
Mario Mikočević
@mozgy
Jun 02 2016 19:35
S S. A P R F .. all you need to know :)
Martin Ayotte
@martinayotte
Jun 02 2016 19:38
Here is a portion of tcpdump (not the whole thing) : https://gist.github.com/martinayotte/62272fd1b8b3471218a5aa2013642967
@mozgy , I don't catch it ...
Me No Dev
@me-no-dev
Jun 02 2016 19:40
what is running on port 8266?
Mario Mikočević
@mozgy
Jun 02 2016 19:40
@martinayotte those are abrevs for tcp packet types, S == Syn, S. == SynAck and so on
Martin Ayotte
@martinayotte
Jun 02 2016 19:41
@me-no-dev , yes, I didn't filter anything : telnet, http, ota/mdns
Me No Dev
@me-no-dev
Jun 02 2016 19:42
if OTA is properly set, those connections to port 8266 should not exist... anyway, those are most of the packets here
Martin Ayotte
@martinayotte
Jun 02 2016 19:43
Open the link on gist, you will see further below the start of the html from nytimes.com
I've copied/pasted about half of the full capture
Mario Mikočević
@mozgy
Jun 02 2016 19:43
what is on port 7 ?
Martin Ayotte
@martinayotte
Jun 02 2016 19:44
Port 7, I think it maybe part of mdns, right ?
Me No Dev
@me-no-dev
Jun 02 2016 19:45
no
really strange...
you can see the delay happens when that connection is tried to be opened
also I see the data going to telnet, but not the data from the site... hard to filter :)
Vicne
@vicnevicne
Jun 02 2016 19:47
7 is the standard for echo I think... Weird...
Martin Ayotte
@martinayotte
Jun 02 2016 19:47
Yes, but who is using echo ?
Mario Mikočević
@mozgy
Jun 02 2016 19:48
'''
echo 7/tcp
echo 7/udp
yep
Me No Dev
@me-no-dev
Jun 02 2016 19:49
da hell is there an echo :D
Martin Ayotte
@martinayotte
Jun 02 2016 19:49
I means the only services in my sketch are telnet, http, ota/mdns
Me No Dev
@me-no-dev
Jun 02 2016 19:49
but the delay is always when that happens
Martin Ayotte
@martinayotte
Jun 02 2016 19:49
what do you means ?
Me No Dev
@me-no-dev
Jun 02 2016 19:50
you can see the packets going to the telnet client and the client acking them one after another untill there is a connection to port 7 on the esp
then you get 2-5 seconds delay
Mario Mikočević
@mozgy
Jun 02 2016 19:50
yep -> 10.111.111.11.59571 > 10.111.111.60.7: Flags [S], cksum 0xf353 (incorrect -> 0x796b)
Me No Dev
@me-no-dev
Jun 02 2016 19:50
and it kepps going till the next connection to port 7
Martin Ayotte
@martinayotte
Jun 02 2016 19:52
I'm not following. I still see the delay in the telnet client while doing the capture, yes.
Me No Dev
@me-no-dev
Jun 02 2016 19:55
do you see that it happens only when your computer is trying to connect to the ESP on port 7?
I use tcpdump -nnvvvXXSs 1514 to run tcpdump and it shous a bit more info (at least on my computer) like the app that is sending the packet
Mario Mikočević
@mozgy
Jun 02 2016 19:59
@martinayotte @me-no-dev I don't see ACKs after SYNs on that dump for connection PC -> ESP on ports 7 and 8266
Me No Dev
@me-no-dev
Jun 02 2016 19:59
the ESP rejects the connection
there will be no ACK :)
Martin Ayotte
@martinayotte
Jun 02 2016 20:00
I've not aware which apps is using the port 7
Mario Mikočević
@mozgy
Jun 02 2016 20:00
exactly, so why is PC so persistent on those ports ?
Martin Ayotte
@martinayotte
Jun 02 2016 20:00
maybe avahi ?
Me No Dev
@me-no-dev
Jun 02 2016 20:00
try running tcpdump with the arguments above (add host filter to the end)
Martin Ayotte
@martinayotte
Jun 02 2016 20:01
Ok !
Me No Dev
@me-no-dev
Jun 02 2016 20:01
then the mumbo-jumbo after the two text lines should include some more stuff
Mario Mikočević
@mozgy
Jun 02 2016 20:01
and host 10.111.111.60
Martin Ayotte
@martinayotte
Jun 02 2016 20:03
Yes ! Of course. Same Gist has been updated !
Me No Dev
@me-no-dev
Jun 02 2016 20:04
does not have that info here :(
Martin Ayotte
@martinayotte
Jun 02 2016 20:07
What do you mean ? I used "sudo tcpdump -i eth0 -nnvvvXXSs 1514 host 10.111.111.60"
Me No Dev
@me-no-dev
Jun 02 2016 20:07
yes command is good, just do not see that extra info I was hoping for
I see something else though
Martin Ayotte
@martinayotte
Jun 02 2016 20:08
:-(
something else relevant ?
Me No Dev
@me-no-dev
Jun 02 2016 20:10
from line 18 to line 150 the server is trying to send the same packet (HTTP[GET] ... code...)
the computer is sending an ack, but the ESP does not hear it
so it resends it a few times untill it finally hears the ack
in a mean time there are a few attempts to port 7 and port 8266
11 seconds you can not get a simple debug message
Martin Ayotte
@martinayotte
Jun 02 2016 20:12
Yes, you're right ! (about port 7 and 8266, maybe it is the pseudo-avahi built in the ArduinoIDE, I will close it and redo capture)
Yes, port 7/8266 was the IDE ... Same Gist updated !
Me No Dev
@me-no-dev
Jun 02 2016 20:16
I bet that at the same time nyt is trying to send you the webpage
and the ESP is returning crap on that end too
Martin Ayotte
@martinayotte
Jun 02 2016 20:17
I don't know, the content displayed on telnet client is consistent, but maybe you're right and many ACK from ESP are crappy
Which isn't the case when using Serial.
Me No Dev
@me-no-dev
Jun 02 2016 20:18
you are keeping the network busy
serial is not keeping it busy :)
so you queue too many packets and the stack can not take it anymore :)
I will mention async again ;)
Martin Ayotte
@martinayotte
Jun 02 2016 20:20
;-)
But the whole thing was not even for my project, I was trying to help this guys on the forum who wish to do a proxy, Serial for him will be useless :-(
Vicne
@vicnevicne
Jun 02 2016 20:22
Yep I am the guy in question and very grateful :-)
Me No Dev
@me-no-dev
Jun 02 2016 20:22
yes I know :) I saw the discussion
Martin Ayotte
@martinayotte
Jun 02 2016 20:22
Hi @vicnevicne
Vicne
@vicnevicne
Jun 02 2016 20:22
Hi @martinayotte
Me No Dev
@me-no-dev
Jun 02 2016 20:23
if it's async, there will be no wait and data will probably be sent with the same or faster rate to the client
Vicne
@vicnevicne
Jun 02 2016 20:23
I am willing to try the async way but am missing pointers on the "async client" part. async server looks OK.
Me No Dev
@me-no-dev
Jun 02 2016 20:23
it's quite more complex code though
Martin Ayotte
@martinayotte
Jun 02 2016 20:24
@me-no-dev is the client on your github too ?
Me No Dev
@me-no-dev
Jun 02 2016 20:24
AsyncTCP holds client and server (and some variations of the client)
Vicne
@vicnevicne
Jun 02 2016 20:24
Well I read the server samples and it looks quite straightforward. But I maybe it gets harder to interleave it with a client...
Me No Dev
@me-no-dev
Jun 02 2016 20:25
@vicnevicne you need to remember a few things when dealing with Async
and I know I lack examples
when you are a client and you want to connecto to a server you use connect and then get callback onConnect or onError
Vicne
@vicnevicne
Jun 02 2016 20:26
I'll be glad to contribute back clean code as an example if I succeed in writing that proxy of course.
Me No Dev
@me-no-dev
Jun 02 2016 20:27
if you get onConnect, then you are connected and you can write(buffer, up to client->space() bytes)
then you will get onAck for each packet that was sent (first send will be a copuple of packets at the same time if you have lots of data)
in onAck you can send another chunk the same way
you get data from the other end in onData
onPoll is something that get's called every 125/256 ms while the connection is open and no data is passing through (in case you have not had data to send on the last onAck)
Vicne
@vicnevicne
Jun 02 2016 20:29
OK. So as all the code will be event driven, I guess the tricky thing will be to have them work in sync
Me No Dev
@me-no-dev
Jun 02 2016 20:29
yes
Vicne
@vicnevicne
Jun 02 2016 20:29
"sync the asyncs", sort of
Martin Ayotte
@martinayotte
Jun 02 2016 20:30
:-)
Me No Dev
@me-no-dev
Jun 02 2016 20:31
the best way maybe is to have cbufs chained containing the dtaa from the nyt site and send those to the telent on it's onAck/onPoll
Martin Ayotte
@martinayotte
Jun 02 2016 20:31
Telnet was for my own easy testing. I presume @vicnevicne wish HTTP Client
Vicne
@vicnevicne
Jun 02 2016 20:32
Yep, but in case the browser doesn't read fast enough, I guess we have to slow down to prevent the chained cbufs from eating all memory...
Martin Ayotte
@martinayotte
Jun 02 2016 20:32
Probably connecting NYTime on STA and providing HTTPServer on AP, right @vicnevicne ?
Me No Dev
@me-no-dev
Jun 02 2016 20:32
using one Async server on port 80 and one async client for each request
he can maybe get 2-3 requests at the same time
Vicne
@vicnevicne
Jun 02 2016 20:34
Err I intended to have the ESP only in STA mode... What would be the advantage of using dual mode STA + AP ?
Me No Dev
@me-no-dev
Jun 02 2016 20:34
I can remember there was a way to make all ESP AP clients go to the ESP web by default for every domain
then a true proxy can be made
@vicnevicne what is your end goal?
Vicne
@vicnevicne
Jun 02 2016 20:35
The goal is to circumvent cross site scripting protection. Use case is this:
Say I have the ESP driving a display from a RSS feed
Except I want a browser to connect to the ESP and select which items of the feed to display or not
So I'd connect to the ESP, and get a page allowing me to select items.
Me No Dev
@me-no-dev
Jun 02 2016 20:37
maybe something like this:
you download the RSS to SPIFFS
use the web server to display a page with the items
Vicne
@vicnevicne
Jun 02 2016 20:37
The issue is that the page from http://esp/... cannot parse data from http://rss_source/... as they are on different server
Well, in my case it's not a RSS feed but a JSON api, but you get the point
Me No Dev
@me-no-dev
Jun 02 2016 20:38
send a request to the ESP with a list of the items and store them (or store a new feed or other format)
then display that
Vicne
@vicnevicne
Jun 02 2016 20:38
Hmmm caching data to the SPIFFS ? Interesing approach indeed.
Me No Dev
@me-no-dev
Jun 02 2016 20:38
json is even better
with JS you can construct the new json and save it for reading from the LCD
you do not do any cross site stuff
you can even check every once in a while if the feed has changed and update only when so
Vicne
@vicnevicne
Jun 02 2016 20:40
Yep. I could also manage the cache duration and avoid re-fetching the data over and over needlessly...
Never worked with SPIFFS, but would be interesting to learn.
Me No Dev
@me-no-dev
Jun 02 2016 20:41
it's quite simple really :)
Vicne
@vicnevicne
Jun 02 2016 20:41
Now that way looks much simpler indeed, avoids concurrent connections, and will be more performant in the end... but I won't contribute the async example I promised you ;-)
Me No Dev
@me-no-dev
Jun 02 2016 20:41
you can check the FSBrowser example
if you want to use sync
else my AsyncWebServer has example with that as well
do not worry for the example :) it's something I should really do... at some point
Vicne
@vicnevicne
Jun 02 2016 20:43
OK. I think I have two tracks I can follow now. Will try the SPIFFS way first this week-end and report back on the thread.
Me No Dev
@me-no-dev
Jun 02 2016 20:43
you need a tool for the IDE so you can upload files to the ESP's SPIFFS
Vicne
@vicnevicne
Jun 02 2016 20:44
Yep read that
Vicne
@vicnevicne
Jun 02 2016 20:44
But in this case I need the file to be live, coming from the web server
Me No Dev
@me-no-dev
Jun 02 2016 20:44
FSBrowser and AsyncFSBrowser both include the needed files
Martin Ayotte
@martinayotte
Jun 02 2016 20:44
Or you can add another web page for upload. it is what I did
Vicne
@vicnevicne
Jun 02 2016 20:46
Mmh, smart indeed. So you can change the webpage to be from the http interface itself ? Cool.
/to be/to be served/
Me No Dev
@me-no-dev
Jun 02 2016 20:46
both examples include an editor so you can start easy with the files
it should edit json files as well
Vicne
@vicnevicne
Jun 02 2016 20:48
Well the json will have to be fetched live upon the first request and downloaded to SPIFFS before it is returned to the browser. Upon subsequent requests in a given timeframe, the cached json will be returned , after which a new version will have to be fetched to replace the previous one. Sounds cool...
Me No Dev
@me-no-dev
Jun 02 2016 20:49
spiffs does not have info like last modified date for files
Vicne
@vicnevicne
Jun 02 2016 20:50
Yes but I can keep that in ram
Me No Dev
@me-no-dev
Jun 02 2016 20:50
but when you are asking for the feed from the remote site, it will probably have it in the header (and there are ways to check just that with HTTP)
Martin Ayotte
@martinayotte
Jun 02 2016 20:50
You can save timestamp in separate file
Me No Dev
@me-no-dev
Jun 02 2016 20:50
point is not to overwrite the file on the FS unless it's new
the flash chip has about 100K write/erase cycles
Martin Ayotte
@martinayotte
Jun 02 2016 20:51
for flash endurance, it can be workarounded by using a SPI FRAM such FM25V10
Me No Dev
@me-no-dev
Jun 02 2016 20:52
or SD card :P
Martin Ayotte
@martinayotte
Jun 02 2016 20:52
Sure ! :-P
Plenty of space !
Vicne
@vicnevicne
Jun 02 2016 20:53
:-) Good to know, but for my usage, I don't intend to do so many cycles, and just keeping a count of number of millis since last fetch will probably suffice.
Me No Dev
@me-no-dev
Jun 02 2016 20:53
first partition should be 3GB FAT
time of last fetch should be in ram
timestamp of the last RSS should be in flash
Vicne
@vicnevicne
Jun 02 2016 20:57
OK guys, I don't want to eat more of your time. thanks very much for your insightful help. Great to follow a discussion between knowledgeable people like you. I had no idea what tcpdump could do, what avahi was, and that we could use STA+AP together just one hour ago. I will report back my progress (and summarize Me No Dev tips for async) in the forum thread. Thanks again !
Me No Dev
@me-no-dev
Jun 02 2016 20:58
good luck with the project :)
Martin Ayotte
@martinayotte
Jun 02 2016 21:04
Welcome @vicnevicne ! And Thanks @me-no-dev (I'm still interested by the AsyncClient ;-) )
Me No Dev
@me-no-dev
Jun 02 2016 21:28
@martinayotte maybe discuss it private so we do not flood the room?
Martin Ayotte
@martinayotte
Jun 02 2016 23:00
@me-no-dev , of course ! probably tomorrow since I'm leaving soon for souper.