What are the technical reasons we have to decide at compile time which modules we want to include?
Could we move towards something like Dynamic Kernel Module Support?
Maybe have a directory
What would it take to also make the firmware provide an additional USB endpoint that offers the flash file system as a mass storage device?
Could we move towards something like Dynamic Kernel Module Support?
Maybe have a directory
modules/ in the flash file system and at boot time inject modules/*.so into the firmware memory? (I'd like it to ignore subdirs, reasons on demand.)What would it take to also make the firmware provide an additional USB endpoint that offers the flash file system as a mass storage device?
Originally I meant using the USB port that I use to power it and upload the firmware. My board looks very similar to this: http://www.nodemcu.com/images/thumbnail/z1s_1.jpg_450x300.jpg but if we can afford the resources to run a second USB device personality, it would sound interesting to do that. I do have a 2 port logic level converter circuit board.
were your "obvious" reasons meant for doing the DKMS compile on the MCU itself? yeah that was so obviously wasteful I didn't even consider it as the goal. :-) I meant it just as a direction, and I also now I see it was a misleading one because turns out you don't even need DKMS just to load kernel modules at runtime, sorry. I had just that latter part in mind, loading pre-compiled additional drivers from the flash disk at boot, so toggling a vendor-specific module would be just rename a file and reboot.
I think it’s not possible to mount it as disk trough uart, thats why I wrote “somehow". Software like nodemcu-uploader, nodemcu-tool exists for exact this purpose - to upload file into spiffs. With Lua it’s a bit different development metodology, usually you create “bones" - firmware with modules predefined modules which you cant replace with Lua code, and you modify / update just Lua parts trough cable / air. All goes fast and painless. I trully understand what your point is, but I doubt that even ecosystems like arduino / mbed have this “modules / dependencies on the fly” feature.
I'm trying however to make the build process as simple as possible, so people can just get started and postpone as many decisions as possible. especially don't have to worry about backupping their flash-stored files just to try a new module for whatever latest blinkentoy vendor.
ideally we could get down to one firmware image per CPU architecture and number type (int/float) and people with more flash will just have more flash storage, then put the optional modules there.
thinking of it, that would even enable me to change modules via wifi, that would be a LOT cooler.
think of this - how many times you need to remove / add module ads1115.c from firmware?
x^n times less than updating your lua code, which you can update in any way you like
HTTP_OTA Lua module as example will take your LFS and update it for you, you choose how to launch it
it’s for ESP8266 mainly because of ram
they are working on the direction to replace as many C modules as possible to Lua ones
from what I see
transfering most modules to LUA sounds like a good idea independently. still, wouldn't it be great to have the capability to change selection of C modules after firmware compile time? would you consider it useless? would it be useful but way to hard to do?
if I tried myself, could I start by just adding a buffer overflow vuln and then have some exploit framework inject the module code, or is a firmware fundamentally different in that attack vector?
oh I see that would not survive reboots.
if it works as planned, people could even download it via wifi without reboot and without requiring flash file memory. (unloading and cleanup might be trickier, dunno.)
Research notes in case someone else wants to continue later:
- dlopen(3) is the C function to load a shared object from a file system.
dlmopencan provide a custom namespace.fdlopenif available can at least load from a file descriptor instead of requiring the library to have a path on a file system, so someone here used it to read from a memory-backed file, and there's a proof of concept project to demonstrate that the file burden is (as expected) entirely a historical choice. - Example: Using dlopen for plugins via dlfcn, example on MacOS but dlfcn seems to be supported by LUA
- Introduction to the types of C libraries. Sounds like for dynamic loading we'll consume extra RAM for pointers to where in RAM the library functions are.
There is a partition table containing entries for SPIFFS, firmware, LFS and others. As long as your new firmware fits in the existing partition the files on SPIFFS will survive.
oh did you mean that any C code loaded has to go to RAM, whereas LUA code can reside on flash, and thus LUA code will always be cheaper than a C module?
in that case it might be worth trying to invest a few of the precious RAM bytes in a swap mechanism.
… which probably won't work without hardware support like a Memory Management Unit.
LFS play kind a swap (read-only) role for Lua. About
C modules, still, it’s easy to build firmware and easy to reload it. You barely need to change C modules at all to be honest. On my daily test unit I have 35 C modules enabled and I still get more than 32K free ram on latest dev. Firmware compile / chip erase / firmware write / LFS load / init.lua is a one-liner bash script.
ok then I'm probably overthinking it and we should just rewrite the beginner's guide some time to make it sound less complicated. after all, having to decide the module selection upfront, before I had any idea about MCUs at all, was what had turned me away at first. only came back because MicroPython still doesn't have the features I want.
back then especially because I had no way to export/import my module selection in the cloud build service to incrementally refine my selection, I'd have had to fill the entire form again for each attempt. at least that's going to be easier soon.
Indeed I am a "linux guy", but also a user experience activist. I already made the Github Action that powers https://github.com/mk-pmb/nodemcu-firmware-daily-vanilla/ to make at least that level of ease available to all Githubbers independent of their OS preference. and I'm planning on building a noob-friendly module selection wizard as well.
when you say the toolchain doesn't support dynamic libs, does it mean there's no known compiler to make shared object files in the first place?