Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Chris
    @cmidgley

    Great review - thanks Peter! Right off the bat I have a couple questions, but may have more as I work on this.

    1) Logically I see why Peripheral is the right base class as it's a device on top of other interfaces, but from an interface perspective it aligns with IO better. Peripheral doesn't have read/write (so those are extensions) and instead expects Sample to be used.

    2) Breaking up the options on the constructor makes sense - I'll pull out the SPI pins so they can live in device.io.SPI (clock, in, out, ...) which I think was your intent as well. I'll need Heltec_Wifi_Kit_32 defined for the device.io.* object (or perhaps we create a new one for Heltec_Wifi_LoRa_32_V2?). What's the best way to go about that? Here are the Heltec Wifi LoRa 32 V2 pins

    3) configure vs. get/set - Set is used in the Serial driver so that's what I adopted as it matched the best. Perhaps Peripheral.configure is the better interface, in which case I'd drop get as I only added it to comply with how Serial did it. If using IO base class, maybe get/set is better and if using Peripheral configure is?

    4) read - I'd like to keep pre-allocated buffers for my use case, but can implement read(number) which returns a dynamically allocated buffer up to max(number) bytes.

    Chris
    @cmidgley

    Update to the above:

    2) Ignore my pull-out of SPI, I see that is "lora" in your example. But since that is SPI, where do I put radio configuration settings like bandwidth, codingRate, frequency, etc? Should "lora" be renamed to something like "interface" and then "lora" have configuration settings? Likewise, where does "onReadable"/"onWritable" go - at the root of the object?

    Chris
    @cmidgley
    For Write, you suggested eliminating the size parameter (and on read to always just dynamically allocate). I've always tried to eliminate dynamic allocations as much as possible in embedded systems as it increases risk of fragmentation and failure on long running systems. Are you not concerned about this with XS?
    Chris
    @cmidgley

    All recommended changes are made, but will further alter based on any feedback from above. Constructor dictionary has sections lora/interrupt/reset as recommended, all other values sit at the root of the dictionary. Now uses configure() instead of set() but still has get(). Close is now safe to call multiple times. Read supports read(buffer) as well as read(number). Write is no longer blocks as is the same for sync/async. typeof undefined changed as recommended. Also extended the library with several new features for the ST127x chip (such as more statistics, gain control, and small packet / high speed chip optimization).

    Here is the latest version as well as a crappy test main to run it (uses build parameter "transmit=0" to build a receive node, and "transmit=1" to build a transmit node). Note: If there is a better way to convert strings to/from ArrayBuffer (see main.js) I'm all ears!

    FYI: Small bug in xsbug when running multiple devices. First machine launches fine in a tab. Second machine starts serial2xsbug and the output window shows interspersed output from both devices. Switching between tabs cleans up the output to be per-device again.
    Peter Hoddie
    @phoddie

    Changes look good. Just a few quick comments:

    • Agreed that Lora is more IO than Peripheral. I think configure makes sense here, since that's exactly the behavior you implemented. get/set is specific for Serial and named to match Web Serial, but I can see how that would be unclear from just the spec.
    • I think write call should use size instead of buf.byteLength when looping to send bytes to the FIFO.
    • I got it wrong when suggesting lora for the property name. The Microchip digital expander is a good example - either spi or i2c is specified depending on the interface. Even with only one supported interface here, I think would make sense to follow here too, so spi instead or lora.
    • This undefined !== typeof options.enableReceiver doesn't avoid the string compare (but make it harder to see!). undefined !== options.enableReceiver does.

    I'm really looking forward to trying this out.

    Peter Hoddie
    @phoddie
    On garbage collection, I'm 1001% with you -- I try to keep that to a minimum. But.... some GC is inevitable and XS is more than fast enough, especially with the small heap sizes on a microcontroller. So, I try to focus on the places that run often. The Ecma-419 APIs are designed to work either way -- using a provided buffer or allocating one -- to be convenient to use when not worrying about GC and efficient when you are. It does put some burden on the implementation of the classes to support that though.
    Peter Hoddie
    @phoddie

    FYI: Small bug in xsbug when running multiple devices. First machine launches fine in a tab. Second machine starts serial2xsbug and the output window shows interspersed output from both devices. Switching between tabs cleans up the output to be per-device again.

    That's a feature, not a bug, When a machine tab is selected, it shows that tab's output. When nothing is selected, it shows them all (merged in time order). That let's you watch several machines at once.

    Chris
    @cmidgley
    All above makes sense. I'll send over an update likely later tonight. I didn't mean to leave the "typeof" there ... that was a global search/replace screw-up! Anyway - fixes seem to be: but in write (size), switch from "lora" to "spi", and remove the "typeof". Trivial - but always feel free to recommend other changes. At some point, if you want, it may make sense to move it into the Moddable tree. I have it in a github repo, but it's a private one along with a bunch of other (currently closed) code.
    Andy Carle
    @PrototypingAndy_twitter

    @bburky Thank you for the M5Paper suggestion above! I can confirm that that looks a lot better when switching between apps on my M5Paper device.

    @phoddie and I have been iterating a bit on the tidiest way to implement this idea in the examples. I think we've settled on this:

    class AppBehavior extends Behavior {
        onDisplaying(application) {
            screen.refresh?.();
            screen.configure?.({updateMode: config.firstDrawMode ?? config.updateMode});
            if (config.firstDrawMode)
                application.defer("onFinishedFirstDraw", config.updateMode);
        }
        onFinishedFirstDraw(application, mode) {
            screen.configure({updateMode: mode});
        }

    That allows for setting both firstDrawMode and updateMode per-device in the manifest and also works in the simulator. I'm going to go ahead and make that change in the examples and add a note to our M5Paper documentation.

    Chris
    @cmidgley
    @phoddie Here are those tiny changes as discussed.
    Peter Hoddie
    @phoddie
    Thanks, @cmidgley! I'll wait to dive into that until I have the hardware. That will give me a better perspective than scanning in an editor.
    Chris
    @cmidgley
    Can you confirm if my understanding is correct that only one mod (.xsa/archive, which might contain multiple modules) can be loaded on a host? I don't see much API (other than xsMachine.archive having a unique archive pointer per machine) to support otherwise.
    Peter Hoddie
    @phoddie
    Correct, there is a single mod. It can contain multiple modules and resources.
    Chris
    @cmidgley
    Question on JSON.stringify with newlines in strings. If I say JSON.stringify('test\n') I get "test\n" from Moddable, but I get "test\\n" (with the newline encoded) from my Chrome browser. I need it to be encoded, so trying to understand if this is expected (and I need to write some encoder that works on the messages) or if it's perhaps a bug?
    Chris
    @cmidgley
    Also in a quick hack to do encoding is failing with an XS abort: dead strip! error (windows simulator). Code that fails is:
    function testEncode(string) {
      return string.replace(/([\n"\&\r\t\b\f])/g, "\\$1");
    }
    Peter Hoddie
    @phoddie

    Is that just how Chrome outputs string in the console? I tried this:

    let foo = JSON.stringify('test\n');
    for (let i = 0; i < foo.length; i++)
        trace(foo.charCodeAt(i), " ", foo[i], "\n");

    And the output seems correct?

    34 "
    116 t
    101 e
    115 s
    116 t
    92 \
    110 n
    34  "
    Peter Hoddie
    @phoddie
    (V8 outputs the same sequence of character codes for the above code)
    Chris
    @cmidgley
    I'll check deeper, but I am having a problem where encoding doesn't work with newlines in strings but does work with characters like quotes.

    FYI - small mistake on XS Mods page - says to add Modules to your manifest:

    "include": [
        $(MODULES)/module/manifest.json
    ]

    But it's actually

    "include": [
        $(MODULES)/base/modules/manifest.json
    ]
    Peter Hoddie
    @phoddie
    Thanks for the mod doc fix. Will correct that. If you can provide an example of where newlines aren't serializing correctly, we'll take a look. FWIW - XS currently passes all the JSON tests in Test262 except the super-obscure replace-function-object-deleted-property test. That isn't definitive -- Test262 doesn't hit everything -- but it is usually a good sign.
    Peter Hoddie
    @phoddie
    @cmidgley - My LoRa boards arrived! I set one up and tried your app in both receive and transmit modes. The runtime resource use looks very reasonable. One small issue -- at some point it looks like the onWritable callback support fell out: #writeCompletion is called when the interrupt fires but there's no code to call onWritable. (I'll set up the other board Wednesday to try transmit & receive together).
    Chris
    @cmidgley

    @phoddie Doh! Fixed ... and I tested callbacks a bit and found several bugs (which required reworking receive). That's what I get for not investing in a test harness (which I still haven't done...!)

    I also changed my buffer management - before I was directly access ArrayBuffer, such as:

    let s = "my message";
    let x = new ArrayBuffer(s.length);
    for (let i = 0; i < x.byteLength; ++i)
       x[i] = s.charCodeAt(i);

    However, I found this incompatible with ArrayBuffer.fromString("my messsage") - each element of x[i] would be undefined. That surprised me ... I could understand byte alignment issues or weird data, but undefined doesn't make sense on an array that has data).

    To fix it, I added a view on the buffer, using Uint8Array. The code works, but now I have a dynamic allocation on every transmit and receive (new Uint8Array(buf)). Any ideas how to eliminate this dynamic allocation? I think it's the only allocation that occurs (when using static buffers).

    Gists are updated for Lora and main.

    Chris
    @cmidgley
    (please ignore the JSON encoding issue I reported; XS is correct)
    Peter Hoddie
    @phoddie
    Thanks for the callback changes. I'll run that in a bit.Glad JSON isn't an issue. On accessing ArrayBuffers, page 107 of our book talks about the inability to access the elements of an ArrayBuffer directly. It is part of the language, for better or worse. Eliminating GC completely from buffer access can be tricky, as a result. Let me see what I can do now that I can run that code.
    Peter Hoddie
    @phoddie
    Got your LoRa example working! Both transmit and receive. It looks there's a bug with buffer re-use. Let the transmitter send up to #35. Then restart the transmitter. The next message received is reported as #15 followed by #25, #35, etc.
    Chris
    @cmidgley
    I'll look into that bug in a bit. One thing ... using onWritable is likely an inappropriate use case for most implementations on LoRa because the normal thing to do is to use it to send another packet, but LoRa FCC (and other country) regulations do not allow that. My main test code violates FCC rules as it sends once/second, exceeding airtime rules.
    Peter Hoddie
    @phoddie
    I imagine the onWritable could be useful if you wanted to put the MCU into deep sleep for a while to save power. Knowing exactly when the packet has gone out would let the power down happen as soon as possible. But, I'm guessing based on experience with other packet based communication because LoRa is all new to me.
    Chris
    @cmidgley
    That's a valid use case for sure.
    When you call close it puts the radio in sleep, so that is what you would want to do prior to going to sleep to further reduce power.
    Chris
    @cmidgley
    What's the requirements with version compatibility of Moddable builds between base firmware and mods? I've been struggling with a mod failure for a couple days, and finally figured out it works when I build everything on my Windows machine, but fails when I take a mod from my Linux machine (my primary use case is mixed systems). The error is when booting, "Mod failed: fatal". The only difference in the two generated files is Windows has 0x0b04 at offset 0x10 (right after VERS) and Linux has 0x0b05 (in the xsa file). I assume that means my Linux system is running a more recent version of Moddable (likely, as it runs in an auto-generated Docker container so currently stays quite fresh). Is it accurate that base firmware and mods must run the exact same version of Moddable?
    Peter Hoddie
    @phoddie
    They need to use the same major version of XS. That because the byte-code changes over time and is not guaranteed to be backwards compatible. The mods documentation briefly notes that ("the XS byte code generated is for a specific version of the XS engine"). Perhaps that should be expanded?
    Chris
    @cmidgley
    Was there such a change recently? How would I check the version / be aware of when it changes? Is there a label used in github when it changes? Trying to figure out how to manage this issue in production.
    Chris
    @cmidgley
    The LoRa buffer issue appears to be just a test code issue. The ArrayBuffer is pre-allocated at 250 bytes. The data arrives and I use String.fromArrayBuffer(buffer)to get the string and print it. ArrayBuffer.byteLength will still be 250 bytes, and as long as the buffer is fresh it works great - but variable size messages will cause this issue. What's best practice here - I see from your book they default to zeros (like calloc), so should I add a trailing null byte to the buffer in the lora code (assuming space available in the buffer) for safety? Or is this just left to the caller to deal with, which if using String.fromArrayBuffer(buffer) likely means an extra dynamic allocation to use something like String.fromArrayBuffer(buffer.slice(0, buffer.byteLength))?
    Peter Hoddie
    @phoddie
    XS major version bumps are roughly once a year. The definitive source of truth for that is in xsCommon.h.
    Chris
    @cmidgley
    I see MAJOR, MINOR and PATCH in there. Feels like in my case I have MAJOR 11 for both, but MINOR 4 on Windows (installed maybe a couple weeks ago?) and MINOR 5 on Linux (installed on a regular basis). Since mod failed with a MINOR change, how often does that happen? Or is that not expected with a MINOR change?
    Also is the version number available in the JS space - or should I implement a simple C module to bring it forward?
    Peter Hoddie
    @phoddie
    The version number is not provided to JavaScript (the language has long tried to avoid that). My mistake - the major and minor versions need to match. Apologies. The patch version doesn't need to match. (The minor version has changed recently)
    On the buffer, your code (read and write) look reasonable. But the result is wrong. I need to explore more, but am out of time at the moment.
    Chris
    @cmidgley

    I think it's easy to explain (come back later and read this). In my main for receive I allocate an ArrayBuffer(250) once, and share it across many reads. When LoRa get 10 bytes, it puts 10 bytes into the ArrayBuffer and returns the size (10). Buffer size is still 250, but the last 240 bytes are all zeros. Using String.fromArrayBuffer works because the rest of the buffer is nulls.

    Later we might receive 5 bytes. Those transfer into the first five bytes of the ArrayBuffer and 5 is returned - all correct. But the buffer still have the prior bytes, so the first 10 bytes remain with values before the null is found.
    When we use String.fromArrayBuffer, there is no other length to use so we get all 10 bytes in the resulting string. I don't believe we can say String.fromArrayBuffer(buffer, size), as they would provide the size hint instead of depending on null/length.

    Therefore this issue is about using String.fromArrayBuffer and not so much in LoRa itself, though we might be able to help avoid issues if we append a null after receiving a packet (assuming space available in the shared buffer). Of note, if dynamically allocated buffers are used for each read this would not occur.

    Sorry for so much texting here ... but on the topic of MAJOR/MINOR ... what pattern do you expect developers to use with customers when consuming mods that customers implement? Do developers tell users exactly which version of Moddable they can use, and then when they OTA/ship a new firmware they somehow coordinate with users to upgrade their Moddable and and install their mods (mods will fail as soon as the new firmware installs)? Or is there some implied backwards compatibility (in my case it was the opposite, I had 11/4 in base and 11/5 in mod)? Could 11/5 in base still run 11/4? If yes, what about 11/5 base running a mod from 10/xxx?
    Peter Hoddie
    @phoddie
    After an update to a newer version of XS, the mod needs to be reinstalled. An incompatible mod will not be loaded, so it the host runs without a mod. The details of that vary by product. Happy to work through scenarios but there's not a single solution that works in all cases. (As noted above, major and minor versions must match -- it doesn't matter whether the host or mod is ahead.)
    Chris
    @cmidgley

    Thanks, Peter, I understand now. Just to give me a sense on how to proceed knowing this, how important has it been, historically, to stay up to date w/Moddable releases? Do you report on critical vs. functional changes (or is that just the release notes and making making our own decision on the criticality of the changes)? If you know, how many critical (security, reliability) required updates have occurred in the last couple years?

    Also - is the Moddable SDK linked to XS for releases, meaning can I take the SDK updates/fixes and not XS? I would guess most security / reliability issues are in the SDK (but perhaps I'm wrong on that?)

    Chris
    @cmidgley
    I added the null byte at the end of the received packet to read() (if buffer has extra room). Solved the problem as expected. If a different solution makes better sense, glad to change. Gist here
    Peter Hoddie
    @phoddie
    @cmidgley - The buffer problems were mostly due to what looks like a mismatch between the test app and driver I was using. I've done some work to clean that up and create an esp32/heltec_lora_32 build target that packages things as expected.
    @cmidgley - You also need to keep in mind that mods need to be reinstall for other reasons - like symbol table changes. More or less that means that you to do some work whenever you push a firmware update. This topic is really a bit much to try to address in Gitter (at least for me).
    Chris
    @cmidgley

    So let me try to summarize best practice. Is it:

    1) Ensure host and mod use the exact same Moddable release (or, more precisely, the same major/minor XS)
    2) Any update to host requires that the Mod is compiled/loaded again (using the same release as the host)

    (and Happy Thanksgiving!)

    Peter Hoddie
    @phoddie
    (Happy Thanksgiving) (1) yes, (2) if the XS versions are the same, then it can be a reload (which could be done from an on-device copy) rather than a rebuild.
    I've been experimenting with some ideas in LoRa to simpify the code. In applying that, it looks like #setLdoFlag may have some mistakes. I'm not sure because I don't really know what it does, but these lines look suspicious:
    let ldoOn = (symbolDuration > 16) & 0x1; // should be >> ??
    config3 = (config3 && 0xfc) || ldoOn << 3; // should be | ?
    (Oh... maybe the first one is OK but just really weird?)