SmingHub/Sming

Sming - Open Source framework for high efficiency native ESP8266 development

slaff

@slaff

It is relatively easy. You should attach your stream to the response. The important part for your stream is to have the available() method to return -1.

And your stream can directly extend IDataSourceStream.

vesley

@vesley

Do you mean I have to implement my own stream which enriches the IDataSourceStream? Isn't possible to use the ChunkedStream class?

Mateusz Zakarczemny

@Matzz

Hi @/all !

I would like to share with you a project winch I was developing for quite some time - Wifire.
It is a Sming framework based ESP application skeleton. It remove form you the hassle of implementing frontend and backend for ESP chip configuration. I created it to be an extensible starting point ESP projects based on Sming framework.

Main features are:

• Rich fronend AJAX based architecture
• WiFi Station and Access Point configuration
• User management with roles based authentication
• GPIO configuration
• GPIO pins switch with timeouts
• Over-the-air updates
• Json based configuration
• Console management (basic)
• React extensible based fronend

Project page - https://github.com/Matzz/Wifire
Known issues and roadmap - https://github.com/Matzz/Wifire/issues

Preview

Some screenshots - https://github.com/Matzz/Wifire/blob/master/docs/preview.md
Signin
User edit
Status
Edit GPIO

I hope that you will like it. Right now I’m developing it alone, so all help and feedback would be helpful.

Best regards,
Matzz

Mike

@mikee47

@vesley The ChunkedStream class is used internally by HttpResponse to perform the actual chunked encoding. As slaff says, all you need to do is provide an IDataSourceStream returning variable-length data by returning -1 from available().

I've roughed-out a template for how you can do this here. There's a more complex example of this in Core/Data/Stream/SectionStream.h.

If you'd like to share more details about how your data is being generated happy to provide further advice.

/*
 * Basic mechanism for using variable-length stream
 */
class MyDataStream : public IDataSourceStream
{
public:
    MyDataStream(/* ... */)
    {
        // ...
    }

    ~MyDataStream()
    {
        // ...
    }

    bool isValid() const override
    {
        // Return false to indicate failure, e.g. attempt to open file in constructor failed
        return true;
    }

    uint16_t readMemoryBlock(char* data, int bufSize) override
    {
        /*
         * Return some data.
         * You are allowed to return data of length 0 here if further processing
         * is required.
         *
         * IMPORTANT! Calling this method more than once should return EXACTLY the same data.
         * The 'stream pointer' must not change.
         */
        if(availableBufferedData == 0) {
            /*
             * Fill buffers etc. Depending on your design, this could also be done in seek()
             */
            getSomeMoreData();
            if(availableBufferedData == 0) {
                allDataSent = true;
                return 0;
            }
        }
        auto size = std::min(bufSize, availableBufferedData);
        memcpy(data, &dataBuffer[bufferPos], size);
        return size;
    }

    bool seek(int len)
    {
        /*
         * Advance the read pointer here.
         */
        bufferPos += len;
        availableBufferedData -= len;

        /*
         * This additional step isn't essential but prevents readMemoryBlock() from returning 0,
         * which is inefficient.
         */
        if(availableBufferedData == 0) {
            getSomeMoreData();
        }
        return true;
    }

    int available() override
    {
        // We don't know final data size: HttpResponse will use chunked transfer mode
        return -1;
    }

    bool isFinished() override
    {
        return allDataSent;
    }

private:
    bool allDataSent{false};
};

void sendSomeChunkedData(HttpResponse& response)
{
    auto stream = new MyDataStream();
    response.sendDataStream(stream);
}

Mike

@mikee47

@rolfrander Try this:

HSPI::Request r1;
r1.setCommand8(0x0c);
r1.in.set32(0); // Setup expected response size (4 bytes)
print_request("*** Request 1", r1);
digitalWrite(PIN_DC, 0);
screen.execute(r1);
print_request("*** Request 1 done", r1);

6 replies

The missing step is r1.in.set32(0) which both initialises the output buffer to 0 and sets the expected response size to 4 bytes.

Checking the datasheet the response to comand 0x0c is only 1 byte, so should be r1.in.set8(0)

vesley

@vesley

Thanks for the example @mikee47. I will take a look after the work. What I'm currently trying (just as an example) is to list all files on the ESP on a page. I'm reading the files on the ESP and want to send in a for loop a chunk per file to the client/browser. I don't have much files, so I could concatenate the list in one string and send it in one shot. But if there are lot of files the string will get huge and this will not work.

Mike

@mikee47

Take a look at the Basic_IFS sample. Does exactly this!

It uses the SectionTemplate class to do the work, all you have to do is provide a template describing how the data should be laid out (text, html, json, etc.)

Mike

@mikee47

vesley

@vesley

I'm not using a template, but could use one. But what if my template data is getting too big? I don't know how the template is internally working, I though it is using placeholders on the template to replace them with data and is sending at the end the content to the client. What will happen if the content is too big so that the memory is not enough?

Mike

@mikee47

SectionTemplate is a relatively new class which extends the standard TemplateStream operation so that the template definition has multiple sections.
For example, the HTML layout in the screenshot is generated from a template with three sections: Header, Content and Footer. The Content section is repeated for each directory record. The data is generated dynamically so RAM usage is low. It doesn't care how big the dataset is.

I modified the sample, as shown below, loading the FWFS partition up with files from ${SMING_HOME}/Libraries/IR/docs/doxygen/html.
There are 698 files in there, totalling 12.3MB (which FWFS compresses down to 2.58M). The listing is 280kB and the ESP8266 has no trouble with that. Total heap usage didn't exceed 20KB.

vesley

@vesley

Great to hear it :). Thank you very much @mikee47 .

Mike

@mikee47

@velsey Welcome! I tried a silly test with 4485 small files pulled from the linux source tree. Listing is 2+MB and other than taking a while to come in worked fine. You can even open files whilst the listing is coming in...

vesley

@vesley

The ESP is more powerful than I though :)

Mike

@mikee47

Just imagine what the ESP32 could do with the brakes off :-)

vesley

@vesley

O yeah :). Is the Sming code for the ESP8266 compatible with the ESP32? Is it enough just to change the host and flash the ESP32?

Mike

@mikee47

It's work in progress but the Basic_IFS sample will run on it.

Mike

@mikee47

Just checked: For the ESP32 you'll need to reduce the app. partition to 960K. (Edit basic_ifs.hw and insert "factory": { "size": "960K" }, into partitions.

slaff

@slaff

Is it enough just to change the host and flash the ESP32?

@vesley This is the final goal. You should compile change only the architecture using SMING_ARCH=Esp32 instead of SMING_ARCH=Esp8266 or Host

Mateusz Zakarczemny

@Matzz

Hi @slaff @mikee47 ,
In AP and Station classes there is a guard before running setIP code which protects us from setting IP outside of init method. Is that needed?
https://github.com/SmingHub/Sming/blob/develop/Sming/Components/Network/Arch/Esp8266/Platform/AccessPointImpl.cpp#L130
https://github.com/SmingHub/Sming/blob/develop/Sming/Components/Network/Arch/Esp8266/Platform/StationImpl.cpp#L203

I did not found any requirement for that in
https://www.espressif.com/sites/default/files/documentation/2c-esp8266_non_os_sdk_api_reference_en.pdf Page 68
or https://www.mikrocontroller.net/attachment/263828/The-ESP8266-Book-August-2015.pdf Page 157

From the other hand there is a note in wifi_get_ip_info to not call it in user_init. But nothing about wifi_set_ip_info

Mike

@mikee47

@Matzz I think you're correct. There's an example in the SDK reference for wifi_softap_set_dhcps_lease (page 65) which actually calls wifi_set_ip_info in user_init(). What happens if you remove the guard? Does anything break? Note: This guard has been present since the original Sming commit!

Mateusz Zakarczemny

@Matzz

Hi,
I'm trying to upgrade my firmware using OTA + sming otaserver. However, while upgrading I'm getting following errors:

0x402593fa: std::function<void (Ssl::Session&, HttpRequest&)>::~function() at /opt/esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/c++/10.2.0/bits/std_function.h:303
 (inlined by) Delegate<void (Ssl::Session&, HttpRequest&)>::~Delegate() at /data/Sming/Sming/Core/Delegate.h:25
 (inlined by) HttpRequest::~HttpRequest() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpRequest.h:78
0x4025effc: HttpClientConnection::~HttpClientConnection() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpClientConnection.h:41 (discriminator 1)
0x4025f02c: HttpClientConnection::~HttpClientConnection() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpClientConnection.h:43
0x4025e33b: String::~String() at /data/Sming/Sming/Wiring/WString.h:207
 (inlined by) ObjectMap<String, HttpClientConnection>::Entry::~Entry() at /data/Sming/Sming/Core/Data/ObjectMap.h:333
0x4025e3a7: Vector<ObjectMap<String, HttpClientConnection>::Entry>::removeElementAt(unsigned int) at /data/Sming/Sming/Wiring/WVector.h:425
 (inlined by) Vector<ObjectMap<String, HttpClientConnection>::Entry>::remove(unsigned int) at /data/Sming/Sming/Wiring/WVector.h:394
 (inlined by) ObjectMap<String, HttpClientConnection>::removeAt(unsigned int) at /data/Sming/Sming/Core/Data/ObjectMap.h:262
 (inlined by) HttpClient::cleanInactive() at /data/Sming/Sming/Components/Network/src/Network/HttpClient.cpp:77
0x40215434: ets_timer_handler_isr at ??:?
0x40215441: ets_timer_handler_isr at ??:?
0x40215486: ets_timer_handler_isr at ??:?
0x40000f65: ?? ??:0
0x40000f49: ?? ??:0
0x40000f49: ?? ??:0


================================================================

And then

================================================================
0x40255849: HttpRequest::reset() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpRequest.cpp:62 (discriminator 1)
0x402593fa: std::function<void (Ssl::Session&, HttpRequest&)>::~function() at /opt/esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/c++/10.2.0/bits/std_function.h:303
 (inlined by) Delegate<void (Ssl::Session&, HttpRequest&)>::~Delegate() at /data/Sming/Sming/Core/Delegate.h:25
 (inlined by) HttpRequest::~HttpRequest() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpRequest.h:78
0x4020b38f: scan_parse_beacon at ??:?
0x4020b3fb: scan_parse_beacon at ??:?
0x4000050c: ?? ??:0
0x402074b5: ieee80211_parse_beacon at ??:?
0x40100ff3: ets_timer_arm_new at ??:?
0x40100ead: ets_timer_disarm at ??:?
0x4000050c: ?? ??:0
0x402470b0: String::invalidate() at /data/Sming/Sming/Wiring/WString.cpp:161
0x40000f68: ?? ??:0
0x40247623: String::operator=(char const*) at /data/Sming/Sming/Wiring/WString.cpp:352
0x40235442: mc_malloc at /data/Sming/Sming/Components/malloc_count/malloc_count.cpp:232
0x40259934: HttpConnection::resetHeaders() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpConnection.cpp:69
0x402593fa: std::function<void (Ssl::Session&, HttpRequest&)>::~function() at /opt/esp-quick-toolchain/xtensa-lx106-elf/xtensa-lx106-elf/include/c++/10.2.0/bits/std_function.h:303
 (inlined by) Delegate<void (Ssl::Session&, HttpRequest&)>::~Delegate() at /data/Sming/Sming/Core/Delegate.h:25
 (inlined by) HttpRequest::~HttpRequest() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpRequest.h:78
0x4025effc: HttpClientConnection::~HttpClientConnection() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpClientConnection.h:41 (discriminator 1)
0x4025f02c: HttpClientConnection::~HttpClientConnection() at /data/Sming/Sming/Components/Network/src/Network/Http/HttpClientConnection.h:43
0x4025e33b: String::~String() at /data/Sming/Sming/Wiring/WString.h:207
 (inlined by) ObjectMap<String, HttpClientConnection>::Entry::~Entry() at /data/Sming/Sming/Core/Data/ObjectMap.h:333
0x4025e3a7: Vector<ObjectMap<String, HttpClientConnection>::Entry>::removeElementAt(unsigned int) at /data/Sming/Sming/Wiring/WVector.h:425
 (inlined by) Vector<ObjectMap<String, HttpClientConnection>::Entry>::remove(unsigned int) at /data/Sming/Sming/Wiring/WVector.h:394
 (inlined by) ObjectMap<String, HttpClientConnection>::removeAt(unsigned int) at /data/Sming/Sming/Core/Data/ObjectMap.h:262
 (inlined by) HttpClient::cleanInactive() at /data/Sming/Sming/Components/Network/src/Network/HttpClient.cpp:77
0x40215434: ets_timer_handler_isr at ??:?
0x40215441: ets_timer_handler_isr at ??:?
0x40215486: ets_timer_handler_isr at ??:?
0x40000f65: ?? ??:0
0x40000f49: ?? ??:0
0x40000f49: ?? ??:0

My OTA upgrader code is here - https://github.com/Matzz/Wifire/blob/master/app/Services/OtaUpdater.cpp
Could I ask for help? Is that wachdog interrupt or sth else?

slaff

@slaff

@Matzz Try the following - compile your application for Host make SMING_ARCH=Host DEBUG_VERBOSE_LEVEL=3 ENABLE_GDB=1. After that enable the networking: https://sming.readthedocs.io/en/latest/arch/host/index.html?highlight=setup-network-linux.sh#networking and run the application: make flash run. You should get a segfault at some point. If you run the same application with valgrind: make valgrind you will get more hints what is the problem and running the application under the debugger https://sming.readthedocs.io/en/latest/arch/host/debugging/index.html will point you exactly where is that happening.

Mateusz Zakarczemny

@Matzz

@slaff Thank you. The main problem was that I tried to delete ota updater after start() method. However, its still not working. After calling start() chip is not doing anything related to update (but its still operational). Does OTA update suppose to work on Host arch? I just trying to figure out where it will be easier to debug.

Mateusz Zakarczemny

@Matzz

After enabling debug on Sming framework level I get:

Updating...
52180257 Download file:
    (0) http://192.168.1.15:9999/rom1.bin -> 108000
52181242 Creating new HttpClientConnection
52181633 HCC::connect: TCP state: -1, isStarted: 0, isActive: 0
52185898 HCC::connect: connecting ...
52189412 TCP 3fff1300 +connection
52192317 TCP 3fff1300 connect to "192.168.1.15:9999"
52198223 TCP connect result: 0
52199810 Download file:
    (0) http://192.168.1.15:9999/spiff_rom.bin -> 300000
52207548 HCC::connect: TCP state: 2, isStarted: 1, isActive: 1
pm open,type:2 0
58684127 TCP 3fff1300 connection error: -8
112181566 Total connections: 1
112181933 Removing stale connection: State: 4, Active: 0, Finished: 0
112182519 TCP 3fff1300 ~connection
172181562 Total connections: 0
232181554 Total connections: 0

slaff

@slaff

@/all Guys, check out the latest addition in Sming: Asynchronous Graphics Library #2357
from @mikee47 ! It is a great way to start adding graphical interface to your Sming applications.It comes also with Async SPI operations and a lot of other improvements.

kmihaylov

@kmihaylov

Hello. With the latest develop branch I'm trying to build Basic_Blink, w/o luck and with the following error:

Copying generated SDK libraries
touch /home/opt/Sming/Sming/out/Esp32/debug/build/esp32/sdk/.complete
/opt/Sming/Sming/Arch/Esp32/Components/esp32/src/startup.cpp:13:10: fatal error: esp_netif.h: No such file or directory
 #include <esp_netif.h>
          ^~~~~~~~~~~~~

slaff

@slaff

I have tried to compile Basic_Blinc for ESP32 a minute ago and all was fine. I guess your ESP-IDF variables need to be set correctly.

Paste here the output from: echo $IDF_PATH and env

kmihaylov

@kmihaylov

@slaff I did a fresh install and now I successfully compile the Basic_Blink example. I did the following:

rm -rf /opt/esp32
rm -rf /opt/esp-idf

With fresh develop branch I ran the install.sh script.

kmihaylov

@kmihaylov

What is the status of SPI with Esp32?

kmihaylov

@kmihaylov

OK I just made it with the standard SPIClass with the SpiPins struct and it works. :)

kmihaylov

@kmihaylov

Hello. I'm using SPI with two ICs. With each of them I do different beginTransaction (they use different SPI modes). In my app I use Serial.systemDebugOutput(true) and my config variable is DEBUG_VERBOSE_LEVEL = 1.

However my app rapidly switches between the two ICs and my debug output is full of Sming/Arch/Esp32/Core/SPI.cpp's

debug_i("[SPI] APB freq = %u, pre = %u, div = %u, target freq = %u, actual = %u", apbFreq, prediv.prescale,
            prediv.divisor, speed.frequency, prediv.freq);

Do I misunderstand how DEBUG_VERBOSE_LEVEL is used (in combination with Serial.setDebugOutput() or there is some other reason that debug_i() is always printed?

Mike

@mikee47

By default, libraries don't get recompiled when DEBUG_VERBOSE_LEVEL changes (to provide a little more control over behaviour). Try make Sming-build all

kmihaylov

@kmihaylov

@mikee47 thanks, I supposed it works the way you told me, but I thought that with some magic it obeys the project's variables.

Wondering if I can change the global SPI instance default pins. I play with ESP32-WROVER and there's some mismatch with the default pins in SPIClass::BusInfo.

If I create another instance, it works well, but for compliance with some external libraries I would like to preserve the use of the global "SPI" object so I wouldn't modify the library to pass a reference to a custom SPI object.

Mike

@mikee47

@kmihaylov I've created a PR #2360, so you should be able to call SPI.setup() to achieve what you need.

kmihaylov

@kmihaylov

Thank you, I'll give it a try.

kmihaylov

@kmihaylov

Hello. With ESP32 I use sensor_vp, sensor_vn and other input only pins. I do not have any issues with these pins, however some error messages are printed on the Serial output:

entry 0x40080618
E (506) gpio: gpio_set_level(215): GPIO output gpio_num error
E (636) gpio: gpio_set_level(215): GPIO output gpio_num error
E (636) gpio: gpio_set_level(215): GPIO output gpio_num error

I also did

    gpio_config_t config;
    config.pin_bit_mask = (1ULL << encoderLeft);
    config.mode = GPIO_MODE_INPUT;
    config.pull_up_en = GPIO_PULLUP_DISABLE;
    config.pull_down_en = GPIO_PULLDOWN_DISABLE;
    config.intr_type = GPIO_INTR_NEGEDGE;
    gpio_config(&config);

which has the same effect.

Mike

@mikee47

Can you be more explicit? Are you calling pinMode, analogRead, etc...

slaff

@slaff

@/all we will release at the end of the month Sming v4.4. Help us find bugs by testing the latest develop version on your devices and report any issues. You can read more about the new features from here: SmingHub/Sming#2298

slaff

@slaff

@/all we have a new release 4.4.0 - https://github.com/SmingHub/Sming/releases/tag/4.4.0 Enjoy.

This release is code-named "M32". Where "M" and "32" stand for high-quality contributions and improved ESP32 support.
Thanks to @mikee47's and his fantastic work.

Some of the highlights in this release:

Improved ESP32 support with support for different ESP32 variants.
Moved from IDF v.4.1 to IDF v.4.3.
Asynchronous Graphics Library.
Support for LittleFS.
Sming Host Emulator: Enabled Host access to physical COM ports.
Sming Host Emulator: allows running the main logic in the Host emulator while running parts of the hardware dependent code directly on a real micro-controller

And a lot more.

vesley

@vesley

Hello @mikee47, could you please explain how the SectionTemplate is working? How has a template to look like? How to replace content in the template with dynamic content? I looked into the Basic_IFS example, but this one is using IFS as file system. How can I use the SectionTemplate with SPIFFS? I'm testing on a WeMos D1 mini. Thanks in advance.

Mike

@mikee47

Hi @vesley.

IFS is a virtual filesystem and the initFileSystem() function demonstrates mounting a SPIFFS volume in a spiffs subdirectory, with a LittleLFS volume in littlefs. It uses a read-only FWFS root filesystem so that critical files are protected from corruption.

To mount a simple SPIFFS volume requires just a spiffs_mount() call. In the sample, edit the fstest() function and comment out initFileSystem();, uncomment spiffs_mount();

If you want to create a default filesystem image, place the files in a project directory (e.g. 'files') and run make SPIFF_FILES=files. This will get written to your device with a make flash. You can flash it separately with make flashpart PART=spiffs0.

The SectionTemplate is currently lacking proper documentation but here's a brief explanation.
The IFS::DirectoryTemplate class inherits from SectionTemplate and provides the data source for the output. It does this by implementing the nextRecord and getValue methods.

The Basic_IFS sample uses one of three template files depending on the requesteed listing format.
If you take a look in resource/listing.html you'll find three separate sections, marked {SECTION} {/SECTION}.
These are indexed starting at #0 and DirectoryTemplate expects this to contain the page header, section 1 the content and 2 the footer.

The nextRecord method is called before a new content record is about to be output. Here's the annotated DirectoryTemplate implementation:

// Return true if we have a new valid record, false if not
bool nextRecord() override
{
    // Content section we fetch the next directory record, if there is one
    if(sectionIndex() == 1) {
        return directory->next();
    }

    // This code emits the header and footer sections exactly once
    // Returning false suppresses their output completely
    return recordIndex() < 0;
}

This sets up the directory information record. The actual directory information is returned from the getValue implementation:

String getValue(const char* name);

Let me know if you have any other questions!

vesley

@vesley

Thanks for the help @mikee47 . It seems that everything that I need is in Sming, I just have to use it :). One more question. I have created a component/library, with some basic web server functionality, which I want to use in several ESP home projects. Is it possible to add the template (listing.html) in the component? So that when I use the component/library it will include the template without the need to add it in every project.

Where communities thrive

People

Repo info

Activity

Main features are:

Preview