These are chat archives for esp8266/Arduino

21st
Jul 2016
Ivan Grokhotkov
@igrr
Jul 21 2016 02:15
@anisimovsergey that's the way SDK handles events. "disconnect" event has semantics of "going into disconnected state", and one can get into "disconnected" state both from "connected" and from "connecting".
Sergey Anisimov
@anisimovsergey
Jul 21 2016 07:56
Hi @igrr, thank you for the explanation. It's ok, I just wonder if there any way to get something like "onConnecting" event too?
Ivan Grokhotkov
@igrr
Jul 21 2016 08:00
could you give me a use case for that?
i can request it to be added to the SDK but i need to understand why it is necessary :)
Sergey Anisimov
@anisimovsergey
Jul 21 2016 08:01
I just want to distinguish all states like: disconnected (no ssid and can't retry), disconnected (idle), connecting, connected (and got IP). Currently I don't know how to distinguish disconnected (idle) and connecting. Although, I can handle this by disabling automatic connection and doing everything by myself but for the events' completeness it would be good to have "onConnecting" IMHO
It's just for better understanding what's going on with the connection
maybe it's not necessary indeed since it is possible to do using manual connection handling on the client side (in my program)
Sergey Anisimov
@anisimovsergey
Jul 21 2016 08:40
@holgerlembke I would prefer more explicit "onConnectionAttemptFailed" or similar to "onDisconnected" which implies that the connection was successfully established at some point.
Holger Lembke
@holgerlembke
Jul 21 2016 08:53
@anisimovsergey I would prefer completeness. If there are states and some states get events, I would expect to give all states an event.
Holger Lembke
@holgerlembke
Jul 21 2016 09:15
and to keep it clearly, I prefer onStateChange(newstate, oldstate).
Sergey Anisimov
@anisimovsergey
Jul 21 2016 09:23
@holgerlembke totally agree, if we've got a state machine, all the events which are triggering the transitions should be reflected in the API, otherwise it's not possible to reflect the whole state of the machine on the client side. It might not be needed in 90% but without them the API is not complete.
Mehrdad K
@mkeyno
Jul 21 2016 09:40
hi guys , is it possible to concurrently use both pre defined SPIFF file system to get benefit of download and upload files and use custom read/wire direct flash access for special use ,
for my project I've suffer from low speed of SPIFF handling and file fragmentation , thus I find these functions to R/W directly to flash
Me No Dev
@me-no-dev
Jul 21 2016 09:41
as long as you use separate space for both, I see no reason
Mehrdad K
@mkeyno
Jul 21 2016 09:41
********************************************************************************************\
  Save data to flash
  \*********************************************************************************************/
void SaveToFlash(int index, byte* memAddress, int datasize)
{
  if (index > 33791) // Limit usable flash area to 32+1k size
  {
    return;
  }
  uint32_t _sector = ((uint32_t)&_SPIFFS_start - 0x40200000) / SPI_FLASH_SEC_SIZE;
  uint8_t* data = new uint8_t[FLASH_EEPROM_SIZE];
  int sectorOffset = index / SPI_FLASH_SEC_SIZE;
  int sectorIndex = index % SPI_FLASH_SEC_SIZE;
  uint8_t* dataIndex = data + sectorIndex;
  _sector += sectorOffset;

  // load entire sector from flash into memory
  noInterrupts();
  spi_flash_read(_sector * SPI_FLASH_SEC_SIZE, reinterpret_cast<uint32_t*>(data), FLASH_EEPROM_SIZE);
  interrupts();

  // store struct into this block
  memcpy(dataIndex, memAddress, datasize);

  noInterrupts();
  // write sector back to flash
  if (spi_flash_erase_sector(_sector) == SPI_FLASH_RESULT_OK)
    if (spi_flash_write(_sector * SPI_FLASH_SEC_SIZE, reinterpret_cast<uint32_t*>(data), FLASH_EEPROM_SIZE) == SPI_FLASH_RESULT_OK)
    {
      //Serial.println("flash save ok");
    }
  interrupts();
  delete [] data;
  String log = F("FLASH: Settings saved");
  addLog(LOG_LEVEL_INFO, log);
}
Me No Dev
@me-no-dev
Jul 21 2016 09:41
there is a config for 4MB flash and 1MB SPIFFS which will give you plenty
Mehrdad K
@mkeyno
Jul 21 2016 09:41
void LoadFromFlash(int index, byte* memAddress, int datasize)
{
  uint32_t _sector = ((uint32_t)&_SPIFFS_start - 0x40200000) / SPI_FLASH_SEC_SIZE;
  uint8_t* data = new uint8_t[FLASH_EEPROM_SIZE];
  int sectorOffset = index / SPI_FLASH_SEC_SIZE;
  int sectorIndex = index % SPI_FLASH_SEC_SIZE;
  uint8_t* dataIndex = data + sectorIndex;
  _sector += sectorOffset;

  // load entire sector from flash into memory
  noInterrupts();
  spi_flash_read(_sector * SPI_FLASH_SEC_SIZE, reinterpret_cast<uint32_t*>(data), FLASH_EEPROM_SIZE);
  interrupts();

  // load struct from this block
  memcpy(memAddress, dataIndex, datasize);
  delete [] data;
}
Me No Dev
@me-no-dev
Jul 21 2016 09:43
as you can see in this code you use the same space as SPIFFS
in this case NO you can not use both
Mehrdad K
@mkeyno
Jul 21 2016 09:43
I've tried to define custom flash script but get compiler error with selected module config
Me No Dev
@me-no-dev
Jul 21 2016 09:44
what module, what config, what script, what error
read the code you pasted above
answers are there
Mehrdad K
@mkeyno
Jul 21 2016 09:46
so if define two divisions for flash memory one for SPIFF and one for raw R/W ,it is possible
Me No Dev
@me-no-dev
Jul 21 2016 09:46
surely it is
why not?
Mehrdad K
@mkeyno
Jul 21 2016 09:48
I'm totally new in this level programming and yet not have success compile however @martinayotte says , he reach to 16MB/sec for raw reading
Me No Dev
@me-no-dev
Jul 21 2016 09:49
maybe you are shooting too high?
i did tell you before that writing to the same place is a bad idea if you do it often
Mehrdad K
@mkeyno
Jul 21 2016 09:52
I don't know , maybe , however you confirmed those function is ready to go and just need handler to them
"writing to the same place "
Me No Dev
@me-no-dev
Jul 21 2016 09:53
yup, you want custom RW FS
which means to me that you will want to write to it
and I have the feeling that yopu will write at the same space
like from the start of that region
which would lead to flash wear and failure
Mehrdad K
@mkeyno
Jul 21 2016 09:55
I don't get your mean , I've just want to write special file in some memory address and read it in whole place
Me No Dev
@me-no-dev
Jul 21 2016 09:55
imagine your hard drive and that you can write a file only 100 000 times at most
once you get close to that number, your disk will fail
Mehrdad K
@mkeyno
Jul 21 2016 09:57
why many writing ? I may have write couple of files in memory , not to many writing but for show images in LED read it many times
Me No Dev
@me-no-dev
Jul 21 2016 09:58
I don't know your code man! I'm guessing and trying to help you not get into trouble
Mehrdad K
@mkeyno
Jul 21 2016 10:00
I'm so sorry for any miss explain but I didn't intend to writing many times of flash , my main problem is to how fast read those file
as SPIFF feature fragment my files , I've decided to use raw R/W
Holger Lembke
@holgerlembke
Jul 21 2016 10:26
@mkeyno interessting discussion! do you have any data about timing differences?
Mehrdad K
@mkeyno
Jul 21 2016 10:27
yes, I post the sketch in my github repo
couple of sketch each one has different timing result , I will post the new sketch about raw R/W as soon get rid of compile error
Clemens Kirchgatterer
@everslick
Jul 21 2016 10:46
@mkeyno you can even point SPIFFS to a different place in flash without support from the 'linker script'. the linker script only defines the addresses. but tak care not to write over anything important. :)
i do this to use the free space on flash that is normally overwritten by OTA updates.
Mehrdad K
@mkeyno
Jul 21 2016 10:49
thanks @everslick but when I trying to define _SPIFFS_start param in my sketch get compiler conflict error for pre set 'linker script'
Clemens Kirchgatterer
@everslick
Jul 21 2016 10:49
extern "C" uint32_t _SPIFFS_start;
extern "C" uint32_t _SPIFFS_end;
extern "C" uint32_t _SPIFFS_page;
extern "C" uint32_t _SPIFFS_block;

bool fs_init(void) {
  uint32_t spiffs, size, start, end, page, block;
  uint32_t used = ESP.getSketchSize();

  spiffs = (uint32_t)(&_SPIFFS_end) - (uint32_t)(&_SPIFFS_start);
  start  = (used + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1));
  end    = (uint32_t) &_SPIFFS_start - 0x40200000;
  page   = (uint32_t) &_SPIFFS_page;
  block  = (uint32_t) &_SPIFFS_block;
  size   = end - start;

  if (size > spiffs) {
    SPIFFS.end(); // free current FS implementation
    SPIFFS = FS(fs::FSImplPtr(new SPIFFSImpl(start, size, page, block, 3)));
  }

  if (SPIFFS.begin()) {
     ....
to give you an idea
i look if the part of flash that is used for OTA is bigger then the part reserved by the linker script. then i use whichever is bigger
Clemens Kirchgatterer
@everslick
Jul 21 2016 10:55
you could just look for some signature in the free part and if it is NOT there kopie your files from SPIFFS to that free part. then read from there.
if you make an OTA update, your signature will be overwritten, so on the next boot you will again copy your files from SPIFFS to the free flash.
so you have the best of both worlds. SPIFFS for easy use and raw flash reads for performance
(assuming you did your benchmarking on raw flash reads, to see if it is worth it)
Mehrdad K
@mkeyno
Jul 21 2016 11:00
thanks dear @everslick , if I get the right point you create space with this code after assure it is safe to creation SPIFFS = FS(fs::FSImplPtr(new SPIFFSImpl(start, size, page, block, 3)));
is it possible to make space with size of one whole sector to have one big chunk of space
Clemens Kirchgatterer
@everslick
Jul 21 2016 11:05
start and size can be whatever you want
(maybe it must be alligned, though)
but in your case, i would not mess with the SPIFFS itself. just fid out where the 'unused' flash is and use that as your raw read cache
for you the 'free' flash is start = (used + FLASH_SECTOR_SIZE - 1) & (~(FLASH_SECTOR_SIZE - 1)); end = (uint32_t) &_SPIFFS_start - 0x40200000;
Clemens Kirchgatterer
@everslick
Jul 21 2016 11:11
(corrected)
between start and end you can safely store your images
after an OTA update you have to store them again
Mehrdad K
@mkeyno
Jul 21 2016 11:13
@everslick , one of the low speed reading problem was file fragmentation , and should find the way to upload files as single piece , that's why I've tried to get rid of SPIFF block ,page,.. definition and when I could assign part of memory to such purpose I could reach the highest reading ratio
Clemens Kirchgatterer
@everslick
Jul 21 2016 11:15
you shukd have all the bits now to test it
Martin Ayotte
@martinayotte
Jul 21 2016 12:33
@mkeyno, @me-no-dev , @everslick , I like the idea of copying files from SPIFFS into raw block when magic signature is overwritten ... ;-)
Me No Dev
@me-no-dev
Jul 21 2016 12:37
?
what magic signature?
copied where?
Martin Ayotte
@martinayotte
Jul 21 2016 12:56
the discussion above between everslick and mkeyno : his file will be copied from SPIFFS into free raw space below SPIFFS only when a magic isn't found there, then after copy, raw read can be done from there.
Mehrdad K
@mkeyno
Jul 21 2016 13:03
@martinayotte , it is possible to define custom page and block size
Me No Dev
@me-no-dev
Jul 21 2016 13:04
I'm not sure what you mean by "magic" and where will that magic be? who will control that magic?
Martin Ayotte
@martinayotte
Jul 21 2016 13:04
@mkeyno , what do you mean ? For the Raw block ? It is not necessary, simply choose something like, let say, 1M below the SPIFFS.
Me No Dev
@me-no-dev
Jul 21 2016 13:05
@mkeyno I strongly advise you not to touch the block size as that is the fl;ash hardware block size
Martin Ayotte
@martinayotte
Jul 21 2016 13:05
@me-no-dev , simple magic like 0XDEADBEEF at the begining of Raw block to figure out if OTA has overwritten this block
Mehrdad K
@mkeyno
Jul 21 2016 13:05
does those params even necessary for raw R/W ?
Martin Ayotte
@martinayotte
Jul 21 2016 13:06
@mkeyno , the magic that @everslick suggested ? No, only checked once at boot time
Me No Dev
@me-no-dev
Jul 21 2016 13:06
ok I get it now, but why would this need to be implemented in the core?
Martin Ayotte
@martinayotte
Jul 21 2016 13:07
@me-no-dev , not in the core ;-) , in @mkeyno sketch ...
Mehrdad K
@mkeyno
Jul 21 2016 13:12
@everslick suggest to safely open new space with SPIFFS = FS(fs::FSImplPtr(new SPIFFSImpl(start, size, page, block, 3))); code whereas @me-no-dev recommend not touch block size and probably other configuration and @martinayotte suggest it could possible after
ota signature have raw R/W
Me No Dev
@me-no-dev
Jul 21 2016 13:14
well there are so many ways this can go :D
Martin Ayotte
@martinayotte
Jul 21 2016 13:14
Of course :-)
Me No Dev
@me-no-dev
Jul 21 2016 13:14
I also suggested if you have 4MB flash to use the configuration with 1MB SPIFFS (giving you 2.5MB of space to store your custom stuff)
then you can safely start writing at the 1MB mark
have OTA and all else untouched
Martin Ayotte
@martinayotte
Jul 21 2016 13:16
That what I've suggested, but at the 2MB mark, so when sketch is small, even OTA won't overwrite it, right ?
Me No Dev
@me-no-dev
Jul 21 2016 13:16
starting at 1MB will premit OTA of 512KB sketch
plenty if you ask me
but 2MB will also work
Mehrdad K
@mkeyno
Jul 21 2016 13:21
normally I use handleFileUpload() to upload my files in place which is decided by SPIFF, what if I want to store it in custom address , what handler would be look like ?
Me No Dev
@me-no-dev
Jul 21 2016 13:27
you write to the location instead of the file
Martin Ayotte
@martinayotte
Jul 21 2016 13:27
@mkeyno , as @everslick suggested, the upload file will still go into SPIFFS (like in the existing example), but at boot time, you check for magic at the beginning of Raw block, if magic not present, you open the SPIFFS file and copy it into Raw block.
If you write directly into Raw block, it is another possible scenario, but when magic is there, you will have to get the file remotely.
Me No Dev
@me-no-dev
Jul 21 2016 13:30
no need for magic if correct config is used
no need to oiverwrite OTA or anything like that
Mehrdad K
@mkeyno
Jul 21 2016 13:31
so I should normally upload the files and then in custom SPIFF ini copy files in the raw memory and keep the beginning file addresses
Martin Ayotte
@martinayotte
Jul 21 2016 13:32
"custom SPIFFS" ? no need for custom.
Raw block is not a custom SPIFF, it is direct storage using ESP.flashWrite() and ESP.flashRead()
Mehrdad K
@mkeyno
Jul 21 2016 13:33

I mean , custom initializing . is this idea don't mess with @me-no-dev note

yup, you want custom RW FS, which means to me that you will want to write to it
and I have the feeling that yopu will write at the same space
like from the start of that region
which would lead to flash wear and failure

Martin Ayotte
@martinayotte
Jul 21 2016 13:35
But you've answered to that : casual writes/read often
Mehrdad K
@mkeyno
Jul 21 2016 13:36
not often reading , in loop file must be read as 25 times per second
Martin Ayotte
@martinayotte
Jul 21 2016 13:40
The reads won't wear out the flash (it is only the writes that have a life expectancy of about 100,000 cycles)
Mehrdad K
@mkeyno
Jul 21 2016 13:43
ok , feels better , thanks you all @martinayotte @me-no-dev @everslick , I'm gonna after that solution , however I find this method for upload without using SPIFF , you may interested to look at

  void handleFileUpload()
{

  String log = "";
  static byte filetype = 0;
  static byte page = 0;
  static uint8_t* data;
  int uploadSize = 0;
  HTTPUpload& upload = WebServer.upload();

  if (upload.status == UPLOAD_FILE_START)
  {
    String filename = upload.filename;
    filename.toLowerCase();
    data = new uint8_t[FLASH_EEPROM_SIZE];
  }

  if (upload.status == UPLOAD_FILE_WRITE   )
  {
    uploadSize = upload.currentSize;

    int base = 0;

    memcpy((byte*)data + base, upload.buf, upload.currentSize);

      data[upload.currentSize + upload.totalSize] = 0; // eof marker


      byte sectorOffset = 0;

      uint32_t _sector = ((uint32_t)&_SPIFFS_start - 0x40200000) / SPI_FLASH_SEC_SIZE;
      _sector += page / 2;
      _sector += sectorOffset;
      noInterrupts();
      if (spi_flash_erase_sector(_sector) == SPI_FLASH_RESULT_OK)
        if (spi_flash_write(_sector * SPI_FLASH_SEC_SIZE, reinterpret_cast<uint32_t*>(data), FLASH_EEPROM_SIZE) == SPI_FLASH_RESULT_OK)
        {
          //Serial.println("flash save ok");
        }
      interrupts();
      delay(10);
    } 

  if (upload.status == UPLOAD_FILE_END)    delete [] data;

}
Martin Ayotte
@martinayotte
Jul 21 2016 13:46
@me-no-dev , did you saw that post ? http://www.esp8266.com/viewtopic.php?p=51162#p51162
Me No Dev
@me-no-dev
Jul 21 2016 13:52
@martinayotte nope
Martin Ayotte
@martinayotte
Jul 21 2016 14:11
@mkeyno , why are you using an intermediate "data" buffer ?
Mehrdad K
@mkeyno
Jul 21 2016 14:13
that wasn't my function, I've just pasted from easyESP repo,
that repo not use the SPIFF and has it's own method to R/W files
Martin Ayotte
@martinayotte
Jul 21 2016 14:17
I means : doing a memcpy() of upload.buf into data and then call spi_flash_write() with data is a bit useless, spi_flash_write() could use upload.buf directly, avoiding the memcpy()
Me No Dev
@me-no-dev
Jul 21 2016 15:07
This message was deleted
Richard P
@richard-p
Jul 21 2016 18:00
Hey, im trying to upload to my esp-12e using arduino and FTDI, but it isnt working. I am supplying 3v3 from an external supply, GPIO0 is attached to a push button so it is normally HIGH and Reset is the same, GPIO15 is LOW, GPIO2 is HIGH, EN is HIGH . To put it into program mode i press GPIO0 then reset then release reset then GPIO0. When i hit upload most of the time i get the lights flashing but its just the red light on the FTDI and the blue light on the esp and i get the triple combo error (sync, open, upload_mem).
I have already installed the updated esp files from the git