These are chat archives for esp8266/Arduino

5th
Jul 2016
Mehrdad K
@mkeyno
Jul 05 2016 06:44
@igrr , is SPIFF functions inherent from standard stream class or simulated by internal methods
in another way , to what level we can change the function properties and level
Mehrdad K
@mkeyno
Jul 05 2016 06:49
can we manipulate some function in lower level to read faster form SPIFF? for example use inline asm code
Me No Dev
@me-no-dev
Jul 05 2016 06:52
@mkeyno here is the function that reads from SPIFFS: https://github.com/esp8266/Arduino/blob/master/cores/esp8266/spiffs_api.h#L295
as you can see, it calls directly the SPIFFS C API
if you think that SPIFFS is slow, maybe take it to it's own repo
and there is also SD CARD option
not that it's any faster...
Mehrdad K
@mkeyno
Jul 05 2016 06:55
so you say there is not very much option to enhance its feature
pellepl is the guy responsible for port the SPIFF in ESP repo
Me No Dev
@me-no-dev
Jul 05 2016 06:58
here is no port... he wrote the file system and we just attached it to the flash read/write functions
Mehrdad K
@mkeyno
Jul 05 2016 06:58
any way I was hopping one of you guys at least look at my code and isuue , but surprisingly nobody check my issue yet
Me No Dev
@me-no-dev
Jul 05 2016 06:58
running at 160MHz clock, 80MHz QIO flash, I have no speed issues on reading
I'll specially figure out a speed test for you to run
Mehrdad K
@mkeyno
Jul 05 2016 07:00
also I change the speed and it changed but not test QIO mode because I thought it is base on module type and can't change it by software option
what rate did you receive by your test program and specific parameter you set
can you paste your code in the issue thicket
SteveToulouse
@SteveToulouse
Jul 05 2016 07:20
Did you try running the test sketch I posted for you ?
I was away.for a few days so couldn't experiment any further for you.
Mehrdad K
@mkeyno
Jul 05 2016 07:23
not actually because I believed to your result, and base on new tips I change my code and it also successfully but not yet receive to margin that I'm looking for
SteveToulouse
@SteveToulouse
Jul 05 2016 07:23
I tried to change spiffs logical block size at build time by using SPIFFS_SINGLETON but haven't succeeded yet.
OK
The max I got was 9fps
Withe a full "screen" at each frame
Mehrdad K
@mkeyno
Jul 05 2016 07:25
I'm gonna change mode to QID to see what will going , but as you said still far from to assurance reading level
@me-no-dev can you at least tell us which rate did you capture by your test sketch
Me No Dev
@me-no-dev
Jul 05 2016 08:00
@mkeyno here is a test sketch for you:
#include "FS.h"

static const char * testFile = "/test.bin";
static uint8_t * payload;
static uint8_t * buffer;

void writeFile(){
  Serial.println("Write File Test");
  uint32_t startTime = millis();
  File f = SPIFFS.open(testFile, "w");
  if(!f){
    Serial.println("Could not open file for writing");
    return;
  }
  size_t i;
  for(i=0;i<256;i++){
    f.write(payload, 4096);
    optimistic_yield(10000);
    Serial.print(".");
  }
  Serial.println();
  f.close();
  uint32_t timeTaken = millis() - startTime;
  Serial.printf("Writing 1MB took %u ms\n", timeTaken);
}

void readFile(){
  Serial.println("Read File Test");
  uint32_t startTime = millis();
  File f = SPIFFS.open(testFile, "r");
  if(!f){
    Serial.println("Could not open file for reading");
    return;
  }
  size_t i = f.size();
  Serial.printf("File size is %u Bytes\n", i);

  while(i > 4096){
    f.read(buffer, 4096);
    optimistic_yield(10000);
    Serial.print(".");
    i -= 4096;
  }
  f.read(buffer, i);
  Serial.println();
  f.close();
  uint32_t timeTaken = millis() - startTime;
  Serial.printf("Reading took %u ms\n", timeTaken);
}

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  payload = (uint8_t *)malloc(4096);
  if(payload == NULL){
    Serial.println("could not malloc payload");
    while(1) delay(1000);
  }
  memset(payload, 'P', 4096);
  buffer = (uint8_t *)malloc(4096);
  if(buffer == NULL){
    Serial.println("could not malloc buffer");
    while(1) delay(1000);
  }
  memset(buffer, 0, 4096);

  delay(5000);

  SPIFFS.begin();
  Dir dir = SPIFFS.openDir("/"); while (dir.next()) Serial.printf("File: %s %u\n", dir.fileName().c_str(), dir.fileSize());
  Serial.println("format start"); SPIFFS.format(); Serial.println("format end");
}

void loop() {
  writeFile();
  delay(500);
  readFile();
  delay(500);
}
this produced the following results:
format start
format end
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 3400 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 439 ms
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 4615 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 432 ms
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 10814 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 405 ms
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 19058 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 431 ms
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 18894 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 256 ms
Write File Test
................................................................................................................................................................................................................................................................
Writing 1MB took 19309 ms
Read File Test
File size is 1048576 Bytes
...............................................................................................................................................................................................................................................................
Reading took 453 ms
as you can see, writing starts to take longer once all sectors are used, but reading always takes about the same time
Mehrdad K
@mkeyno
Jul 05 2016 08:02
thanks dear @me-no-dev
Me No Dev
@me-no-dev
Jul 05 2016 08:02
and for 1MB file, that is 430ms, or over 2MB per second
Mehrdad K
@mkeyno
Jul 05 2016 08:02
this is with QIO ,80-160Mhz config?
Me No Dev
@me-no-dev
Jul 05 2016 08:03
yes
Mehrdad K
@mkeyno
Jul 05 2016 08:11
thanks , buddy , what exactly optimistic_yield(10000); for ?
Me No Dev
@me-no-dev
Jul 05 2016 08:11
for yielding only when necessary
Mehrdad K
@mkeyno
Jul 05 2016 08:11
so it is not kindad delay function
Me No Dev
@me-no-dev
Jul 05 2016 08:12
it's like delay, just conditional
Mehrdad K
@mkeyno
Jul 05 2016 08:12
ok ,can you do me one more favor , just let me know whether this logic in my loop program is good to with your function or not
Me No Dev
@me-no-dev
Jul 05 2016 08:13
@mkeyno you need to run the sketch above
if it's ok, then you need to look into your sketch
I have no time to read, understand and debug it
please understand
Mehrdad K
@mkeyno
Jul 05 2016 08:14
ok, thanks any way
SteveToulouse
@SteveToulouse
Jul 05 2016 08:19
@me-no-dev your results look pretty much the same as mine. Looks like the best to be hoped for, with standard SPIFFS block/page sizes, is a little over 2MByte/s
I think I was using DIO though (iirc NodeMCU doesn't do QIO by default ?)
Me No Dev
@me-no-dev
Jul 05 2016 08:20
yes, I use custom module
[Read] Start
[Read] size is 2621440 Bytes
[Read] took 998 ms
pretty close to a second. File size is 2.5MB
Mehrdad K
@mkeyno
Jul 05 2016 08:22
custom module means custom setting set ? right?
Me No Dev
@me-no-dev
Jul 05 2016 08:22
means not the original module on my nodemcu
mine is qio
DIO+40MHz flash+80MHz CPU = 2152 ms for the same 2.5MB file
SteveToulouse
@SteveToulouse
Jul 05 2016 08:23
PS the results file I posted includes a 50 microsecond delay to simulate writing to the LEDs. Throughput around 1MB/s which seems logical if the SPI is running half as fast as yours
Me No Dev
@me-no-dev
Jul 05 2016 08:27
QIO vs DIO = 998 vs 1132 ms, so that does not matter much
if your flash chip can run at 80MHz, then that is what you need to get some speed
SteveToulouse
@SteveToulouse
Jul 05 2016 08:29
Yep. I also wanted to try changing the logical page size of spiffs at build time. Maybe make them 1024 instead of 256, to reduce housekeeping overhead (4 times less reads ?).
998 vs 1132, depsite doubling the raw I/O rate, seems to suggest quite a bit of overhead
Me No Dev
@me-no-dev
Jul 05 2016 08:29
would also mean that you will waste more bytes for files under 1K
SteveToulouse
@SteveToulouse
Jul 05 2016 08:30
true re: byte waste but I think @mkeyno's files are minimum 75K, hence the idea
tried building with SPIFFS_SINGLETON
Me No Dev
@me-no-dev
Jul 05 2016 08:31
DIO+40MHz+160MHz = 1457 ms
SteveToulouse
@SteveToulouse
Jul 05 2016 08:32
and changed the page size constant but it doidn't do anything. Will look at the code to see why.
Me No Dev
@me-no-dev
Jul 05 2016 08:33
defined on many places ;) do not remember which one matters most... I think in the linker
SteveToulouse
@SteveToulouse
Jul 05 2016 08:33
ahhhh I thought it was too simple to be true ;-)
Mehrdad K
@mkeyno
Jul 05 2016 08:33
linker?
Me No Dev
@me-no-dev
Jul 05 2016 08:34
do not change page size though, if 1MB/sec is not fast enough on a microcontroller...
@mkeyno area you should not touch
SteveToulouse
@SteveToulouse
Jul 05 2016 08:35
@mykeno is trying to update a "screen" made of LED strips at 25fps
so needs pretty high throughput
and it worked on an Arduino Tiny (iirc)
with and SD card reader
Mehrdad K
@mkeyno
Jul 05 2016 08:36
actually it was teensy3.1 not AVR arduino board
SteveToulouse
@SteveToulouse
Jul 05 2016 08:36
ahhh sorry
itsy bitsy teeny weeny ;-P
Mehrdad K
@mkeyno
Jul 05 2016 08:37
and still suprise how that my clumsy code with SD card work fine with teensy and powerful ESP cant work it fine with higher clock
Me No Dev
@me-no-dev
Jul 05 2016 08:37
have you tried with SD Card?
Mehrdad K
@mkeyno
Jul 05 2016 08:38
off course Ivan said it is due to hardware SPI with DMA feature of teensy
Me No Dev
@me-no-dev
Jul 05 2016 08:38
SPIFFS lives on the same flash as your code! There is lots going on to load the functions for execution, read your file and whatnot
Mehrdad K
@mkeyno
Jul 05 2016 08:38
on teensy yes but no try with ESP
SteveToulouse
@SteveToulouse
Jul 05 2016 08:38
he's probably right
Mehrdad K
@mkeyno
Jul 05 2016 08:40
but I intend to use pretty feature of ESP repo , right know I made websoever that can easily upload the image of SPIFF and set the show priority and duration , pretty good platform to make module for animation show
SteveToulouse
@SteveToulouse
Jul 05 2016 08:42
It looks like you'd save a lot of time and energy by using an SD card. Do you have any free GPIOs after installing the LED strips?
Mehrdad K
@mkeyno
Jul 05 2016 08:43
yes , the strip only use 2 pin and 1 pin for hall sensor (detect revolution)
you suggest if read from SD card probably reach better reading speed
Me No Dev
@me-no-dev
Jul 05 2016 08:45
you can use the same test sketch I gave you with little changes to run for SD
get a card and run the test
Mehrdad K
@mkeyno
Jul 05 2016 08:45
ok,I'm on it
Me No Dev
@me-no-dev
Jul 05 2016 09:16
sdcard at 40MHz (160 CPU, 40 Flash, QIO)
[Read] Start
[Read] size is 2621440 Bytes
[Read] took 4341 ms
so, much slower
Mehrdad K
@mkeyno
Jul 05 2016 09:18
I was thinking the same , thanks to test it before I burn my fingers with Iron solder ,
so we should come back to SPIFF
Me No Dev
@me-no-dev
Jul 05 2016 09:26
given that you acheave max SPIFFS speed of 2.5MB/s, your reading on it's own will let you do only about 33 fps (30ms for 75K file), You add the time it takes to transmit that to your leds for each frame and you will get the time it takes to draw the screen once
that is why video compression exists :) you stream full HD these days with less bandwith
maybe lose some color
SteveToulouse
@SteveToulouse
Jul 05 2016 09:32
Maybe need to forget the ESP01 module @mkeyno . I think you said it was only running at half the speed (40MHz CPU/20MHz SPI iirc)
Mehrdad K
@mkeyno
Jul 05 2016 09:32
if you say 30ms for 75K by SPIFF is totally ok by me ,although my margin is 40ms for each frame , also I might add in POV each frame come by half the total image line , because in each revolution we have 2 frame and half of image line we could reach to 1 frame
I have tested on ESP12F
SteveToulouse
@SteveToulouse
Jul 05 2016 09:33
ok
Mehrdad K
@mkeyno
Jul 05 2016 09:33
but I'm going to run @me-no-dev sketch on the ESP12F
I'm gonna replace my reading function with @me-no-dev function following is my sketch to pars the reading chunk and put it in LED strip
if( (millis()- OpenlastTime) > DURATION[image_index]*1000L )
                                                            {
                                                              Serial.print(F("duration="));  Serial.print(millis()- OpenlastTime);
                                                              FILe.close();
                                                              image_index++; 
                                                              if(image_index>IMAGE_NUM)    image_index=0;

                                                               Serial.print(F(" i="));  Serial.print(image_index); 
                                                              Serial.print(F(" New Image="));  Serial.println(IMAGES[image_index]);
                                                               FILe=SPIFFS.open("/SHOW/"+IMAGES[image_index], "r");
                                                              if(!FILe) Serial.println("FILE NOT OPEN");
                                                              OpenlastTime=millis();
                                                              Current_imageLine = 0;
                                                            }
if(finish_chunk)
                {
                  Serial.println("read chunk");
                 int i = 0; 
                   while( (i < NUM_LEDS*3*9) && FILe.available())
                  { 
                       i += FILe.readBytes( ((char*)_4kbuffer) + i, (NUM_LEDS*3*9)-i);
                    }
                 finish_chunk=false; 
                }

if( (micros()-lastLineShow)> lineInterval)
                                            {

                                              byte *p=&_4kbuffer[L*NUM_LEDS];
                                              memcpy(leds,p,NUM_LEDS);         

                                              FastLED.show(); 
                                              lastLineShow=micros();

                                              Current_imageLine++;  
                                              L++;
                                              if(L>=9) 
                                                    {
                                                      L=0;
                                                      finish_chunk=true;
                                                    }                                      
                                            }
if(Current_imageLine >= IMAGES_LINES)   
                                    { 
                                    Serial.print("[frame]"); Serial.println((micros()-lastFrameShow));
                                    Current_imageLine = 0;
                                    lastFrameShow=micros(); 
                                    Start_new_round=false; 
                                    FILe.seek(0, SeekSet);

                                    }
SteveToulouse
@SteveToulouse
Jul 05 2016 09:49
@me-no-dev seems like a pretty slow SD card you used? Or something else limiting? Even old class 2 cards get 2MByte/s.
Me No Dev
@me-no-dev
Jul 05 2016 09:50
@SteveToulouse at 80MHz SPI: [Read] took 4061 ms
can't go much higher
SteveToulouse
@SteveToulouse
Jul 05 2016 09:53
I would have expected to see at least the same speed as on-board flash
cos' not sharing the bus with code fetches
Me No Dev
@me-no-dev
Jul 05 2016 09:55
4 times less bandwith and the lib is from the AVR world, slightly adapted to ESP
SteveToulouse
@SteveToulouse
Jul 05 2016 09:55
ahh OK
Me No Dev
@me-no-dev
Jul 05 2016 09:55
also no idea what overhead the FAT FS gives on it's own
SteveToulouse
@SteveToulouse
Jul 05 2016 09:56
true. should be less when reading contiguous blocks though ecause FAT doesn't store housekeeping data in the blocks, unlike SPIFFS
or maybe it's implementation dependant
I mean, FAT doesn't need to store housekeeping in the data blocks - it's all in the reserved FAT sectors at the start of the FS
Clemens Kirchgatterer
@everslick
Jul 05 2016 15:54
This message was deleted
reserve() is mor or less useless, because concat() and = will mess up the buffer anyway, won't it?
i thought, i just have to reserve() my expected amount of bytes and String will use that buffer as long as it is big enought.
but as soon as i assign or concat() something to the string, it does realloc() the buffer.
This message was deleted
Clemens Kirchgatterer
@everslick
Jul 05 2016 16:02
hmm. icould be wrong though
unsigned char String::reserve(unsigned int size) { if(buffer && capacity >= size) return 1;
reserve() does nothing if buffer is big enough.