embedded-javascript/moddable

Discuss using the Moddable SDK

Chris

@cmidgley

Slight update to script... this one is working. Is this the best approach for now?

rm -rf build
mkdir build
mcrun -p esp32/heltec_wifi_kit_32 -o build
pushd build/tmp/esp32/release/mod
sed -i '/$(SERIAL2XSBUG) $(UPLOAD_PORT) $(DEBUGGER_SPEED) 8N1 -install $(ARCHIVE)/d' makefile
make
popd

Peter Hoddie

@phoddie

For consistency, the right thing would be to implement -d for mcrun. I don't think that should be too difficult (but I'm notorious bad with make scripts). Calling serial2xsbug directly is fine.

Chris

@cmidgley

Do you mean -t? (for -t build)? -d is implemented - to trigger xsbug at least (doesn't do anything else as I can see)

You may like this... I now have a Google Cloud Run container that has Moddable, with a REST API. You just call the endpoint like this:

{
    "config": {"mySetting": "one", "anotherSetting": "two"},
    "script": "console.log('test');\nconsole.log('two');"
}

And then Cloud Run launches a Docker contains with a REST API that returns something like:

{
    "compileErrors": [],
    "binary": "AAABxVhTX0EAAAAMVkVSUwsFAAAAAAAYU0lHThJR/B2lKKlari/PPUNF7JQAAAAYQ0hLUwAAAAAAAAAAAAAAAAAAAAAAAAAeTkFNRXRlY2gubW9kZGFibGUuYnVpbGQtMAAAAAAvU1lNQgcAQABsb2cAY29uc29sZQBjb25maWcAZGVmYXVsdABvbmUAdHdvAAAAASxNT0RTAAAAEFBBVEhtb2QueHNiAAAAAEFDT0RFi1cAADEsDACRAgBmAgBCYAEAKMkFdGVzdACpAY+RAgBmAgBCYAEAKMkEdHdvAKkBj0RIj28Ce6FEAAAAElBBVEhjaGVjay54c2IAAAAAVkNPREVXAAAxFAwAmwKjAjgEADEFDADdj0ThAo9ESI9XAAAxBwwAmwKjAkRIj88DAMkKbWMvY29uZmlnAM8EAG8D2c8EAIuLzwQAbwTZbwR7oUQAAAAXUEFUSG1vZC9jb25maWcueHNiAAAAAFRDT0RFi1cAADE0DACbAqMBiI2zAlwCyQl0ZXN0LW9uZQCGBQBvAFwCyQl0ZXN0LXR3bwCGBgBvAN8BNAGPREiPzwQAi4vPBABvBNlvA3uhRAAAAAhSU1JD"
}

or in the case of a compile error:

{
    "compileErrors": [
        "/workspace/builds/build-2/mod.js:2: error: end of file in string!",
        "/workspace/builds/build-2/mod.js:2: error: missing ,!",
        "/workspace/builds/build-2/mod.js:2: error: missing )!"
    ]
}

And thanks to Google Run pricing, it costs only $3/month to run 30,000 compiles, complete with geographic distribution (runs closest to the user) and with scale up/down on demand. Pretty cool!

Blake Burkhart

@bburky

I've been enjoying playing with Moddable over the weekend, it seems nice overall. I wanted an easy to use framework to play with the M5Paper I got recently.

One issue with switching to fast refresh modes like A2, is it the full screen may not be cleared on startup and filled areas are poorly drawn. It was even noticeable on the epaper-flashcards demo, old content from other demos wasn't cleared off the screen.

I ended up with this code which seems to work well. First, clear the screen. Then draw content on screen with a high quality mode to draw all the UI well. Then defer changing the refresh mode to something like A2 (it must be deferred, or the mode changes before the UI is initially drawn).

class AppBehavior extends Behavior {
    onDisplaying(application) {
        if (screen.configure && screen.refresh) {
            screen.configure({updateMode: "GC16"}); // Any high quality mode
            screen.refresh();  // Use INIT mode to clear the screen fully (optional, and slow. It causes a double flash, but is needed to clear all ghosting)
            application.defer("finishedOnDisplaying")  // Must be deferred so that the screen can finish drawing
        }
    }
    finishedOnDisplaying() {
        screen.configure({updateMode: "A2"});
    }
}

Is this the best way to do this? You may want to update some of the examples to do this if so, this draws with much better quality. Is application.defer() the best way to run something after UI has finished rendering?

Alternately, something like a "temporarily use this mode for the next update" command could achieve this too, but I couldn't see how to do that.

Peter Hoddie

@phoddie

@bburky - Thanks for giving the Moddable SDK a try. On the M5 Paper unit that I used to implement the ePaper display driver (and that @lprader used to implement the example apps) we didn't see the visual glitch you describe. We do have one report of it happening on a different M5 Paper unit though, so perhaps there is some variation or something we've overlooked. The approach you took looks pretty reasonable as a solution to the problem you describe. The finishedOnDisplaying function should be invoked after both the refresh and first frame update are complete. (The checks for screen.configure and screen.refresh is unnecessary if running on M5 Paper. I'm guessing that's so you can run in the simulator?)

Blake Burkhart

@bburky

Yes that's for the simulator, I think the example code used an if (config.updateMode) check which is about the same.

Peter Hoddie

@phoddie

"temporarily use this mode for the next update" could be done in the IT8951 driver, though that kind of behavior introduces state that could be confusing.

I'm a few miles away from my M5 Paper device at the moment. I'll try this out when I can. If your approach generates better results, we should update the examples (and perhaps the driver?) to implement your behavior.

Blake Burkhart

@bburky

Agreed. Reusing the application events with onDisplaying() and defer() is pretty easy and works. I don't think it needs to be done at the driver level if this covers all use cases.

Epaper is just a bit weird, there are times you want fast refresh and times you want high quality. The Moddable app I made over the weekend was a replacement for a horrible phone app that interfaces with a BLE kitchen scale. A2 refresh does a good job of quickly updating the UI with new live data, but I used the high quality mode at startup to draw the main UI crisply.

Peter Hoddie

@phoddie

Yea, every ePaper display has its own special behavior. It is always a bit of challege, though the results often look really great.

Cool that you were able to get BLE + Piu working together on M5 Paper. Please consider sharing a video, if not the code. I suspect a lot of people could learn from that.

Blake Burkhart

@bburky

I'll think about it, so far I just got the BLE connection and really really simple UI. Needs a few more features, then I'll post it somewhere.

Peter Hoddie

@phoddie

Cool.

Peter Hoddie

@phoddie

@cmidgley - The LoRa driver looks pretty good. You've gotten much of the Ecma-419 stuff right. I put together some notes on where some fine tuning would help to sync it with the intent of the spec. Those are a little long for here, so they are in a gist. I'm happy to try making some of these changes when my boards arrive, so please don't feel obligated. If something isn't clear or doesn't make sense, just let me know.

Chris

@cmidgley

Great review - thanks Peter! Right off the bat I have a couple questions, but may have more as I work on this.

1) Logically I see why Peripheral is the right base class as it's a device on top of other interfaces, but from an interface perspective it aligns with IO better. Peripheral doesn't have read/write (so those are extensions) and instead expects Sample to be used.

2) Breaking up the options on the constructor makes sense - I'll pull out the SPI pins so they can live in device.io.SPI (clock, in, out, ...) which I think was your intent as well. I'll need Heltec_Wifi_Kit_32 defined for the device.io.* object (or perhaps we create a new one for Heltec_Wifi_LoRa_32_V2?). What's the best way to go about that? Here are the Heltec Wifi LoRa 32 V2 pins

3) configure vs. get/set - Set is used in the Serial driver so that's what I adopted as it matched the best. Perhaps Peripheral.configure is the better interface, in which case I'd drop get as I only added it to comply with how Serial did it. If using IO base class, maybe get/set is better and if using Peripheral configure is?

4) read - I'd like to keep pre-allocated buffers for my use case, but can implement read(number) which returns a dynamically allocated buffer up to max(number) bytes.

Chris

@cmidgley

Update to the above:

2) Ignore my pull-out of SPI, I see that is "lora" in your example. But since that is SPI, where do I put radio configuration settings like bandwidth, codingRate, frequency, etc? Should "lora" be renamed to something like "interface" and then "lora" have configuration settings? Likewise, where does "onReadable"/"onWritable" go - at the root of the object?

Chris

@cmidgley

For Write, you suggested eliminating the size parameter (and on read to always just dynamically allocate). I've always tried to eliminate dynamic allocations as much as possible in embedded systems as it increases risk of fragmentation and failure on long running systems. Are you not concerned about this with XS?

Chris

@cmidgley

All recommended changes are made, but will further alter based on any feedback from above. Constructor dictionary has sections lora/interrupt/reset as recommended, all other values sit at the root of the dictionary. Now uses configure() instead of set() but still has get(). Close is now safe to call multiple times. Read supports read(buffer) as well as read(number). Write is no longer blocks as is the same for sync/async. typeof undefined changed as recommended. Also extended the library with several new features for the ST127x chip (such as more statistics, gain control, and small packet / high speed chip optimization).

Here is the latest version as well as a crappy test main to run it (uses build parameter "transmit=0" to build a receive node, and "transmit=1" to build a transmit node). Note: If there is a better way to convert strings to/from ArrayBuffer (see main.js) I'm all ears!

FYI: Small bug in xsbug when running multiple devices. First machine launches fine in a tab. Second machine starts serial2xsbug and the output window shows interspersed output from both devices. Switching between tabs cleans up the output to be per-device again.

Peter Hoddie

@phoddie

Changes look good. Just a few quick comments:

Agreed that Lora is more IO than Peripheral. I think configure makes sense here, since that's exactly the behavior you implemented. get/set is specific for Serial and named to match Web Serial, but I can see how that would be unclear from just the spec.
I think write call should use size instead of buf.byteLength when looping to send bytes to the FIFO.
I got it wrong when suggesting lora for the property name. The Microchip digital expander is a good example - either spi or i2c is specified depending on the interface. Even with only one supported interface here, I think would make sense to follow here too, so spi instead or lora.
This undefined !== typeof options.enableReceiver doesn't avoid the string compare (but make it harder to see!). undefined !== options.enableReceiver does.

I'm really looking forward to trying this out.

Peter Hoddie

@phoddie

On garbage collection, I'm 1001% with you -- I try to keep that to a minimum. But.... some GC is inevitable and XS is more than fast enough, especially with the small heap sizes on a microcontroller. So, I try to focus on the places that run often. The Ecma-419 APIs are designed to work either way -- using a provided buffer or allocating one -- to be convenient to use when not worrying about GC and efficient when you are. It does put some burden on the implementation of the classes to support that though.

Peter Hoddie

@phoddie

FYI: Small bug in xsbug when running multiple devices. First machine launches fine in a tab. Second machine starts serial2xsbug and the output window shows interspersed output from both devices. Switching between tabs cleans up the output to be per-device again.

That's a feature, not a bug, When a machine tab is selected, it shows that tab's output. When nothing is selected, it shows them all (merged in time order). That let's you watch several machines at once.

Chris

@cmidgley

All above makes sense. I'll send over an update likely later tonight. I didn't mean to leave the "typeof" there ... that was a global search/replace screw-up! Anyway - fixes seem to be: but in write (size), switch from "lora" to "spi", and remove the "typeof". Trivial - but always feel free to recommend other changes. At some point, if you want, it may make sense to move it into the Moddable tree. I have it in a github repo, but it's a private one along with a bunch of other (currently closed) code.

Andy Carle

@PrototypingAndy_twitter

@bburky Thank you for the M5Paper suggestion above! I can confirm that that looks a lot better when switching between apps on my M5Paper device.

@phoddie and I have been iterating a bit on the tidiest way to implement this idea in the examples. I think we've settled on this:

class AppBehavior extends Behavior {
    onDisplaying(application) {
        screen.refresh?.();
        screen.configure?.({updateMode: config.firstDrawMode ?? config.updateMode});
        if (config.firstDrawMode)
            application.defer("onFinishedFirstDraw", config.updateMode);
    }
    onFinishedFirstDraw(application, mode) {
        screen.configure({updateMode: mode});
    }

That allows for setting both firstDrawMode and updateMode per-device in the manifest and also works in the simulator. I'm going to go ahead and make that change in the examples and add a note to our M5Paper documentation.

Chris

@cmidgley

@phoddie Here are those tiny changes as discussed.

Peter Hoddie

@phoddie

Thanks, @cmidgley! I'll wait to dive into that until I have the hardware. That will give me a better perspective than scanning in an editor.

Chris

@cmidgley

Can you confirm if my understanding is correct that only one mod (.xsa/archive, which might contain multiple modules) can be loaded on a host? I don't see much API (other than xsMachine.archive having a unique archive pointer per machine) to support otherwise.

Peter Hoddie

@phoddie

Correct, there is a single mod. It can contain multiple modules and resources.

Chris

@cmidgley

Question on JSON.stringify with newlines in strings. If I say JSON.stringify('test\n') I get "test\n" from Moddable, but I get "test\\n" (with the newline encoded) from my Chrome browser. I need it to be encoded, so trying to understand if this is expected (and I need to write some encoder that works on the messages) or if it's perhaps a bug?

Chris

@cmidgley

Also in a quick hack to do encoding is failing with an XS abort: dead strip! error (windows simulator). Code that fails is:

function testEncode(string) {
  return string.replace(/([\n"\&\r\t\b\f])/g, "\\$1");
}

Peter Hoddie

@phoddie

Is that just how Chrome outputs string in the console? I tried this:

let foo = JSON.stringify('test\n');
for (let i = 0; i < foo.length; i++)
    trace(foo.charCodeAt(i), " ", foo[i], "\n");

And the output seems correct?

Peter Hoddie

@phoddie

(V8 outputs the same sequence of character codes for the above code)

Chris

@cmidgley

I'll check deeper, but I am having a problem where encoding doesn't work with newlines in strings but does work with characters like quotes.

FYI - small mistake on XS Mods page - says to add Modules to your manifest:

"include": [
    $(MODULES)/module/manifest.json
]

But it's actually

"include": [
    $(MODULES)/base/modules/manifest.json
]

Peter Hoddie

@phoddie

Thanks for the mod doc fix. Will correct that. If you can provide an example of where newlines aren't serializing correctly, we'll take a look. FWIW - XS currently passes all the JSON tests in Test262 except the super-obscure replace-function-object-deleted-property test. That isn't definitive -- Test262 doesn't hit everything -- but it is usually a good sign.

Peter Hoddie

@phoddie

@cmidgley - My LoRa boards arrived! I set one up and tried your app in both receive and transmit modes. The runtime resource use looks very reasonable. One small issue -- at some point it looks like the onWritable callback support fell out: #writeCompletion is called when the interrupt fires but there's no code to call onWritable. (I'll set up the other board Wednesday to try transmit & receive together).

Chris

@cmidgley

@phoddie Doh! Fixed ... and I tested callbacks a bit and found several bugs (which required reworking receive). That's what I get for not investing in a test harness (which I still haven't done...!)

I also changed my buffer management - before I was directly access ArrayBuffer, such as:

let s = "my message";
let x = new ArrayBuffer(s.length);
for (let i = 0; i < x.byteLength; ++i)
   x[i] = s.charCodeAt(i);

However, I found this incompatible with ArrayBuffer.fromString("my messsage") - each element of x[i] would be undefined. That surprised me ... I could understand byte alignment issues or weird data, but undefined doesn't make sense on an array that has data).

To fix it, I added a view on the buffer, using Uint8Array. The code works, but now I have a dynamic allocation on every transmit and receive (new Uint8Array(buf)). Any ideas how to eliminate this dynamic allocation? I think it's the only allocation that occurs (when using static buffers).

Gists are updated for Lora and main.

Chris

@cmidgley

(please ignore the JSON encoding issue I reported; XS is correct)

Peter Hoddie

@phoddie

Thanks for the callback changes. I'll run that in a bit.Glad JSON isn't an issue. On accessing ArrayBuffers, page 107 of our book talks about the inability to access the elements of an ArrayBuffer directly. It is part of the language, for better or worse. Eliminating GC completely from buffer access can be tricky, as a result. Let me see what I can do now that I can run that code.

Peter Hoddie

@phoddie

Got your LoRa example working! Both transmit and receive. It looks there's a bug with buffer re-use. Let the transmitter send up to #35. Then restart the transmitter. The next message received is reported as #15 followed by #25, #35, etc.

Chris

@cmidgley

I'll look into that bug in a bit. One thing ... using onWritable is likely an inappropriate use case for most implementations on LoRa because the normal thing to do is to use it to send another packet, but LoRa FCC (and other country) regulations do not allow that. My main test code violates FCC rules as it sends once/second, exceeding airtime rules.

Peter Hoddie

@phoddie

I imagine the onWritable could be useful if you wanted to put the MCU into deep sleep for a while to save power. Knowing exactly when the packet has gone out would let the power down happen as soon as possible. But, I'm guessing based on experience with other packet based communication because LoRa is all new to me.

Chris

@cmidgley

That's a valid use case for sure.

When you call close it puts the radio in sleep, so that is what you would want to do prior to going to sleep to further reduce power.

Chris

@cmidgley

What's the requirements with version compatibility of Moddable builds between base firmware and mods? I've been struggling with a mod failure for a couple days, and finally figured out it works when I build everything on my Windows machine, but fails when I take a mod from my Linux machine (my primary use case is mixed systems). The error is when booting, "Mod failed: fatal". The only difference in the two generated files is Windows has 0x0b04 at offset 0x10 (right after VERS) and Linux has 0x0b05 (in the xsa file). I assume that means my Linux system is running a more recent version of Moddable (likely, as it runs in an auto-generated Docker container so currently stays quite fresh). Is it accurate that base firmware and mods must run the exact same version of Moddable?

Peter Hoddie

@phoddie

They need to use the same major version of XS. That because the byte-code changes over time and is not guaranteed to be backwards compatible. The mods documentation briefly notes that ("the XS byte code generated is for a specific version of the XS engine"). Perhaps that should be expanded?

Chris

@cmidgley

Was there such a change recently? How would I check the version / be aware of when it changes? Is there a label used in github when it changes? Trying to figure out how to manage this issue in production.

Chris

@cmidgley

The LoRa buffer issue appears to be just a test code issue. The ArrayBuffer is pre-allocated at 250 bytes. The data arrives and I use String.fromArrayBuffer(buffer)to get the string and print it. ArrayBuffer.byteLength will still be 250 bytes, and as long as the buffer is fresh it works great - but variable size messages will cause this issue. What's best practice here - I see from your book they default to zeros (like calloc), so should I add a trailing null byte to the buffer in the lora code (assuming space available in the buffer) for safety? Or is this just left to the caller to deal with, which if using String.fromArrayBuffer(buffer) likely means an extra dynamic allocation to use something like String.fromArrayBuffer(buffer.slice(0, buffer.byteLength))?

Where communities thrive

People

Repo info

Activity