Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Gregor Hartmann
    @HHHartmann
    please make sure to not have a wlan2.lua or wlan2.lc file on your esp8266. Might be that it is prefered over the LFS version.
    Patrick Dorion
    @poorandunlucky
    well i reflashed the firmware, i reflashed the LFS with the dummy strings, i disabled the require () in init.lfs for the original wlan module, and now my RAM runs around 30K with or without the module loaded, but something else is broken in type comparison/logic in a module i haven't touched in a long time, i even reverted it back to an older known to work version and it didn't work... is it supposed to be unstable, i just realized "well it's a $2.50 module from China, maybe it's not super stable and things breaking like that are to be expected"...
    could be ghosts in my computer, too, mind you...
    Patrick Dorion
    @poorandunlucky
    just wanted to say that i just fixed everything, and i'm pretty proud! RAM usage is still kinda high, though, but i'm still hovering around 30 K (37k at boot), so it's still ok...
    thanks for the encouragement 😊
    Patrick Dorion
    @poorandunlucky
    if anyone is bored, or curious, and wants to try or see, it's in https://github.com/poorandunlucky/nodemcu-lfs and you'll need LFS/wlan2.lua LFS/log.lua, and conf/*, run conf/init.conf (for the variables), and then require ( modname ), then it's log.msg ( 'message', 'facility', 'level' ) and wlan2.switch ( 'OFF|STA|AP' ) ...
    Patrick Dorion
    @poorandunlucky
    oh and nothing actually broke, it's log that was buggy, it's actually a wonder it worked in the first place, but i fixed it now... it ended-up not working anymore, but it was never solid, now it should be ok...
    Terry Ellison
    @TerryE
    @poorandunlucky Patrick, I can't see the repo. Is it public?
    Terry Ellison
    @TerryE

    now my RAM runs around 30K...

    How many modules have you got loaded and have you enabled HTTPS? Essentially, if you load everything, then your modules will allocate quite a lot of RAM at boot. My general approach is only to load the modules that my Lua apps are likely to use. I also keep my ESPs to a local control LAN / Wifi and use an RPi as a control bridge. I don't access the public internet (or talk directly to end user devices such as mobiles from my IoT devices. I usually have around 44 Kb free RAM at boot.

    Terry Ellison
    @TerryE

    As a general check I run all my code through luac -l -l -o /dev/null file.lua and check the output; not so much the opcodes generated but scan for globals (_ENV references in Lua 5.3) and track my Upvalue use in my closures. Globals are bad news because they hang around even when you've finished with them. Locals are the cheapest type of variable storage both in terms of runtime and RAM footprint. I usually use outer scoping through Upvalues to make them accessible to inner functions; again a cheap and fast option. The thing about local declarations is that they are allocated on the stack so they only take up a Tvalue (the minimun storage quantum in the Lua runtime), or relocated to a special linked area if closed because an inner function is still live but the outer defining function has completed. All Lua magic that just works; and incidentally works really efficiently. Lua locals are garbage collected once they leave scope.

    Another little trick is if a module contains any one-time initialisation, then wrap any sub functions within its scope and make it ephemeral as follows. Note that this only works if it is one-time. A second call will crater as the local variable referencing now holds nil. If this is compiled into LFS, then all of the code, constants and code metadata are in LFS. Yes it will use some RAM during execution, but this will all be returned to the heap on garbage collection after exiting its scope.

    local function someInit()  -- Upvalues: someInit and what other outer module locals are used
       someInit = nil
       -- other local and local functions
       local function x() --[[  etc ]] end
      -- ...
    end
    Nicol Spies
    @NicolSpies
    @TerryE , welcome back Terry :smile:
    Terry Ellison
    @TerryE
    @NicolSpies, I was really quite ill for a long time: virus triggered reemergence of old ME/CFS symptoms -- sort of long Covid before SARS-COV-2 ever existed. My brain wasn't working well enough to sensibly engage so I just dropped my NodeMCU work and kept up some lighter dev / sysadmin commitments that I have, and that were still within my capability. I am feeling a bit better, so I want to take stock of my contribution to the project and complete or kill some my dangling tasks, really focusing on stabilising outstanding work on the Lua53 core.
    Nicol Spies
    @NicolSpies
    @TerryE , I am really glad to hear that, I feared for the worst, I had it light mid July and still have bouts where coffee tastes like plastic. I thought the NodeMCU momentum was gone for good and decided to embrace the Espressif RTOS, C and JTAG using the ESP32C3 after years of contemplation. Powerful but more complicated than Lua but I enjoy the challenge. Now the candle of hope to have a Lua environment on the ESP32 is flickering again.
    Terry Ellison
    @TerryE
    Part of the issue here was that a lot of our contributers have moved on and I was trying to take on more an more to keep the project going. It just wasn't sustainable, so we need to be far more focused on what we allow as open issues with a general rule that we only support them if a dev resource can be identified. Wishlist and single-user "I can't get my app working" issues don't work.
    I am still using 8266s and Lua. I have an energy-efficient house (passiveHouse-class and performance but not certified), ESPs have been part of my heating, environment and HW control system for 5 years now, though TBH, if I were starting from scratch I would probably just use RPi zeros. :)
    Terry Ellison
    @TerryE
    I've only just got around to reimplementing the Lua apps in Lua5.3 on LFS! The source code is about half as long. :smile:
    Nicol Spies
    @NicolSpies
    I agree fully, what worries me is the lua NodeMCU userbase compared to the ESP-IDF RTOS userbase is shrinking. By moving on, the mover gains and everybody else loose out. Proficiency takes effort and dedication but as have been acknowledged many time on the group, there are things that are beyond the capabilities of the 8266 that is relatively easy on the 32.
    Your 8266 web server springs to mind
    So to jump into the deep end of RTOS and C at the age of 57, I sometimes wonder.
    I think that is the drawing of the 8266 and lua; to do things in a constrained environment that others do not even attempt.
    Terry Ellison
    @TerryE
    The real issue with the ESP8266 and Lua (or microPython) is the lack of RAM. LFS helps a lot. But this makes unrealistic to use RTOS on the 8266. In my case the Wemos D1 ESP32 boards are pretty much module swaps in my existing H/W and have similar pricing. The main user of the ESP8266 chip are the Sonoff and Shelly devices, and I would never consider a NodeMCU firmware for these. Tasmota makes far more sense to me.
    At some point we need to consider moving the ESP8266 branches into frozen support and switch focus to the ESP32 branch.
    Nicol Spies
    @NicolSpies
    Yes, I remember the pre-LFS days when I started with the 8266 starting up with less than 20kb if IRAM. I agree with moving everything over to the ESP32, I believe that is what Johny is working on at $work. It appears as such a different mindset that the risk might be that little of all the 8266 effort will be retained.
    Terry Ellison
    @TerryE
    I am going to support Johny on this.
    Nicol Spies
    @NicolSpies
    Espressif is bringing out new SoCs at such a rate that the 8266 feels like the 4004 processor
    Terry Ellison
    @TerryE
    Yup
    Nicol Spies
    @NicolSpies
    +1 on the support of Johny
    Terry Ellison
    @TerryE
    Or more the 8080 :laughing:
    Nicol Spies
    @NicolSpies
    I suppose the moving across will leave the NON-RTOS behind
    Terry Ellison
    @TerryE
    Yup. Its not so much ESP8266 vs ESP32, but more SDK vs IDE+RTOS is the killer from a support perspective.
    Nicol Spies
    @NicolSpies
    Yes, that is what I am seeing with the new pair of JTAG and RTOS glasses
    So I still believe that for contained environments like your smart-home, the 8266 is perfect, in the runaway stagecoach of cloud IoT it is required to switch just to stay on the board.
    Nicol Spies
    @NicolSpies
    I have to run, nice to chat, cheers
    Terry Ellison
    @TerryE
    Yup enjoy. Busy hosing and killing dead issues. :smiley:
    Nicol Spies
    @NicolSpies
    :thumbsup:
    Patrick Dorion
    @poorandunlucky
    @poorandunlucky Patrick, I can't see the repo. Is it public?

    @TerryE Sorry, I had the repo set to private in case I published my password, which is still a good precaution, so I made a public one... https://github.com/poorandunlucky/lfs-public

    sorry about that 😕

    also very sorry to hear that you were sick, the whole pandemic thing always seems like a real joke until someone you know has complications because of it... some of my mother's friends had problems with it, also, and i didn't know you had it, but i'm glad you pulled through 😊
    Patrick Dorion
    @poorandunlucky
    also copied what you said about globals, and RAM optimization... i didn't know you could get more than around 30K, i load a lot of modules to play with it on my desk like that, but knowing you can get up to 40 comfortably is going to be helpful as a target...
    Terry Ellison
    @TerryE
    Re passwords and other shared secrets, you should never embed them in source that you put in git. A common approach is to have a secrets.lua or secrets.json which you load at startup into local variables in RAM and use .gitexcludeto prevent this file from being added to the repo. However, remember that this will still leave these secrets in a form that can still be read from flash. There are ways to mitigate this but things can get quite complicated to do this properly.
    You need to do a threat assessment, and design your approach according. My preference is isolate my IoT devices on a control LAN interfacing to an RPi which runs a locked- down Linux OS and layer my security over standard Linux stacks.
    I'll take a look at the source.
    Terry Ellison
    @TerryE
    BTW, I've downloaded the ZIP of your repo so you can remove the public attribute
    Terry Ellison
    @TerryE
    @poorandunlucky, your RAM usage is high because the code isn't written with IoT footprint in mind. For example log.luais pretty verbose for what it does uses RAM where there is little to be gained from doing so:
    • module() and package.seeall are Lua 5.0 constructs which are deprecated in 5.1 and run inefficiently on NodeMCU. I am not even sure they work in Lua 5.3.
    • You shouldn't be printing errors -- that's just a good way to miss them in prod systems. Errors are just that and in Lua they should be thrown with an error() statement.
    • In general this code would be fine for running on a real system with lots of RAM, but its not a good fit for a IoT device with ~40Kb RAM. OK, you want a log(message,facility, level ) function and the ability to configure the level from 'DEBUG' to NONE with an option filename, but this IoT functionality doesn't need RAM arrays or this quite verbose logic.
    • for example the strtolvl2()logic could be coded in a dozen lines, run faster and use no RAM arrays:
      level = level:upper()
      if ( level == 'DEBUG' ) then return 7
      -- ...
      elseif level == 'EMERG' or level == 'EMERGENCY'  then return 0
      else error( 'Invalid error level specified: '..level)
      end
    • `lvl2str()2 is just the conjugate, but given that these seem to be private methods anyway so that the API can be sloppy over whether string or numeric log levels would be used, I am not sure if this code is needed.
      I've got other suggestions, but this I am not sure here is the place to do it.
    Terry Ellison
    @TerryE

    BTW, you could also code the str2lvl function something like the following. Perhaps, not the clearest code, but only a 26 instruction function that is fast and has low RAM use in an ESP8266 LFS context. OK, it will miscode error levels like "NOTE", but IMO it does the job. (Note that string.find(s,p,i,true) does a simple fast memory scan.)

    local function str2lvl(s)
      local vals = 'DEB\07INF\6NOT\5WAR\4ERR\3CRI\2ALE\1EME\0'
      local i = vals:find(s:sub(1,3):upper(),1,true) or error('Invalid error level '..s)
      return vals:sub(i+3,i+3):byte()
    end

    Of course there are many ways of implementing this, but some take far less ESP8266 resources than others. All you need is a little lateral thinking in your approach.

    Matthew B White
    @mbwhite

    Hello; just wanted to say thankyou to the hard-work put into this project; I've come back to use the esp8266 that have been lying around not being used for too long.

    Very much enjoying using them. As a professional software engineer by day, I can see the effort that's gone into it :clap:

    rigorka
    @rigorka
    @TerryE Hi, pardon my asking here, but it seems you're the only person who might have a fairly clear idea how/if my problem can be solved. I've had my laptop stolen from me a day after I've written a very large and boring piece of Lua code (SNMP control implementation for some really awkward power management hardware). I did not commit this code to git, so I no longer have the .lua files :( However I still do have the debug esp8266 module where this code has been uploaded after compiling it into LFS image with luac.cross from nodemcu-firmware. I was able to extract the LFS image from esp8266 back to my PC. Now I'd love to give luadec a try to decompile that code, but to do so I need to extract the compiled Lua files (.lc) from that LFS image. Is there any reasonable way to do so?
    Terry Ellison
    @TerryE
    @rigorka, no sorry. Not realistically practical.
    rigorka
    @rigorka
    @TerryE - thanks anyway. At least not gonna waste my time trying, will just rewrite that stuff from scratch.
    Darren Enns
    @darethehair_twitter
    New guy here with a simple/short question: Do either the 'release' or 'dev' versions in the NodeMCU builder produce working LUA firmware for making 'https' calls? Yes, I have added 'tls' options. The docs hint that sites that use CloudFlare to provide HTTPS access may not work with NodeMCUs implemention of TLS. Thanks.