Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Peter Hoddie
    @phoddie
    I understand this code is an excerpt so the big picture is lost. I can't really imagine how to design a solution because I don't understand what this is trying to do.
    Chris
    @cmidgley

    Imagine you have a class with many objects inside it. You want to mock it, so you use a generic mocking service to do it:NewClass = mock(OldClass) (or newObject = mock(new OldClass()). That class that wraps the initial proxy, and as the handler executes it continues to proxy what it finds (get/construct/access).

    No worries - I'll find a way around this. My issue - not Moddable's. Thanks for the help.

    Peter Hoddie
    @phoddie
    The problem is triggered because Timer doesn't call methods of the object, but has static methods (e.g. you call Timer.clear(t) instead of t.clear()). The latter will work through a proxy because the receiver can/will be set correctly.
    If Timer is all that gets in the way (it is a rare API style in the Moddable SDK, mostly to be closer to the web timeout API). you could just make a proxy-friendly implementation of the Timer class.
    Chris
    @cmidgley
    I actually have that, but I need to stop the proxy for recursively doing the Timer itself. Perhaps decorators, perhaps something else. I proved it works though by checking for any property name that contains "timer" and that got my code to work - so now it's just a matter of style and a bit of elbow grease.
    Peter Hoddie
    @phoddie
    You could take inspiration from unscopables and have an unproxyable property to list the property names that shouldn't be proxied. (But even mentioning unscopables in a serious conversation may be grounds for being banished from TC39, so....)
    Chris
    @cmidgley
    I was thinking something like @unmockable with a TS decorator, which would then basically do what unscopables does (if I understand it - I just googled it as I hadn't heard of it before ... but attach a symbol)
    Peter Hoddie
    @phoddie
    With TypeScript + Proxy + Decorators, you are in territory I dare not tread. Too much magic.
    Chris
    @cmidgley
    I respect that. At least this is only used in the test framework, never in the actual product.
    Chris
    @cmidgley
    OK - this is pretty cool... adding a TS decorator to a property executes during preload! Sweet...
    Peter Hoddie
    @phoddie
    Magic. ;)
    Peter Hoddie
    @phoddie
    Coming back around to proxy with host functions, from your example yesterday with Array.prototype.push. Internally to XS there are actually two kinds of host functions – a host object and a host primitive. The host primitive is more compact and less standard. As part of preparing the ROM image, the XS linker converts host objects to host primitives. That step is skipped when strip is disabled (which is why turning off strip allows your Proxy to work). In most places, XS automatically and invisibly promotes a host primitive to a host object when needed (fxToInstance). The special case to do that promotion isn't in the Proxy constructor, so it fails. We'll add that. Just FYI, a more focused workaround is to use Object.assign which promotes the host primitive to a host object: new Proxy(Object.assign([].push), {}).
    Chris
    @cmidgley

    Thanks! Since I can't tell which objects are host, I'd have to do that on every object, property, etc., which I'd prefer not to do. But it sounds like you found the fix?

    I have @unmockable working now and it's resolved my issue with Timer - so I'm all good there! Getting close to have preload working on all modules (one big module left to go). Dependency injection and Mocking are working now (major rework to Mock, a bit of refactoring to Dependency Injection). Hopefully a few more days and I'll see what the new slot footprint looks like.

    Peter Hoddie
    @phoddie
    Yes, we know how to fix it. We'll get that in place. Use whatever workaround you like until then. ;)
    Congrats on the progress of your preload and mock harmonization initiative.
    Chris
    @cmidgley

    Mind if I ask a couple performance/memory questions?

    1) If I have a method that needs to return an object, is there much benefit in slots/memory/fragmentation if I return a native object (return { prop: value }) vs. use a class to contain the object (return newClass(value); where the class provides the prop property)?

    2) Is it correct that if I have an object that is not preloaded with six properties, it will consume 7 slots (1 for the object, and 6 for the properties, or a total of 112 bytes)?

    3) Likewise, if I have an array of 100 elements, it takes 100 slots? (And if the array has objects, we multiple the above with this ... or in this case, 100 elements * 6 slots = 600 slots = 9.6K?)

    4) Would using CDV to encapsulate heavy usage objects (arrays of objects with many properties) drop the slot count to one per CDV buffer?

    Peter Hoddie
    @phoddie
    First, there's no fragmentation. One of the nifty features of the XS allocator and the host implementation on microcontrollers is that neither the slot nor chunk heaps fragment. That's because the slot heap uses a single fixed-size allocation and the chunk heap can be compacted to coalesce free space.
    The most precise answers to your questions can be had by adapting the obscure (and incredibly useful) xsuse example.
    Assigning a value to a property to an object requires a slot. Some values are contained in the property (Boolean, number, null, undefined) others use additional memory (strings use a chunk, for example). The assignment references the existing value in that case, so there's no duplication / cloning.
    compileDataView is great for reducing memory used by the values it can carry. The cost, of course, is they are slower to access - though for many cases that just doesn't matter.
    Peter Hoddie
    @phoddie
    Short answer: use xsuse to measure.
    Chris
    @cmidgley
    Didn't catch xsuse before ... I'll play with it in a few days. Thx.
    Lawrence R Kincheloe
    @LokiMetaSmith

    just pulled the public branch and did the "make install" for linux, got hit with the following errors.
    make install -f simulator.mk
    make[1]: Entering directory '/home/loki/Projects/moddable/build/makefiles/lin'

    simulator debug : cc main.c

    /home/loki/Projects/moddable/build/simulator/lin/main.c: In function ‘onApplicationOpen’:
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:99: warning: ‘.xsa'’ directive output may be truncated writing 5 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:49: note: ‘snprintf’ output between 21 and 8211 bytes into a destination of size 8208
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~~~~~~~~~~~~~~~~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:99: warning: ‘.so'’ directive output may be truncated writing 4 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:49: note: ‘snprintf’ output between 20 and 8210 bytes into a destination of size 8208
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^
    ~~~~~~~~~~~~~~~~~~
    Assembler messages:
    Fatal error: can't create /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o: No such file or directory
    make[1]: [simulator.mk:59: /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o] Error 1
    make[1]: Leaving directory '/home/loki/Projects/moddable/build/makefiles/lin'
    make:
    [makefile:66: install] Error 2

    Michael Kellner
    @mkellner
    @LokiMetaSmith, please "make" before "make install -f simulator.mk"
    Peter Hoddie
    @phoddie

    @cmidgley – a fix the issue you ran into, being unable to create a Proxy for a host function primitive, will be pushed in a couple days. Meanwhile, if you want to give it a try before that, in xsProxy.c after line 780 add these lines:

    #ifdef mxHostFunctionPrimitive
        if ((mxArgc > 0) && (mxArgv(0)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(0));
        if ((mxArgc > 1) && (mxArgv(1)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(1));
    #endif

    There's a corresponding change for Proxy.revocable, but it doesn't sound like you are using that (yet!).

    Chris
    @cmidgley
    Thanks. I don't yet see a need for revocable, but good to know.
    Peter Hoddie
    @phoddie
    Just a matter of time. ;)
    Chris
    @cmidgley
    Feels that way!
    Lawrence R Kincheloe
    @LokiMetaSmith

    @LokiMetaSmith, please "make" before "make install -f simulator.mk"

    yep, that's my issue!

    Chris
    @cmidgley

    Using Instrumentation to get live info on slots - I'm finding the doc and the first example are wrong, as the numbers are not correct (for example, doc says 11 is SlotHeapSize yet it is actually 14 on ESP32). I see in xsuse it scans to find the correct number - that's how I got to 14. What's the story here?

    Also, the type definition for it is very difficult to use. The type InstrumentationOffset is not exported, and really only limits a numeric value and enforces it to be 1-16 (even though different numbers are used). And you can't use the actual names as they are only types and not exported constants. Same issue with WiFi (and perhaps others) on their type definitions. Just an FYI for now, but I think the fix (if instrumentation numbers are static and don't need to be scanned) is to export constants from the .js file and declare them in the .d.ts file (perhaps using a enum, not sure yet). If instrumentation numbers are dynamic, I've got no good ideas for typing ... maybe a dictionary lookup method in .js that does the scan and return the correct number for the instrumentation?

    Peter Hoddie
    @phoddie
    The offsets are different for each port. xsuse was working correctly on ESP32, ESP8266, and macOS very recently. Can take another look.
    Peter Hoddie
    @phoddie
    Just ran xsuse on macOS, ESP8266, and ESP32. All results look reasonable. Here's the start of output from ESP32:
    Boolean: 0 bytes
    Number: 0 bytes
    String `String in ROM`: 0 bytes
    String `fromCharCode(32)`: 12 chunk bytes = 12 bytes
    String `String in ` + 'RAM': 24 chunk bytes = 24 bytes
    Object {}: 1 slots = 16 bytes
    Object {x: 1}: 2 slots = 32 bytes
    Object {x: 1, y: 2}: 3 slots = 48 bytes
    Object {x: 1, y: 2, z: 3}: 4 slots = 64 bytes
    Date: 2 slots = 32 bytes
    BigInt 1n: 12 chunk bytes = 12 bytes
    BigInt 100000000001n: 16 chunk bytes = 16 bytes
    Function () => {}: 7 slots = 112 bytes
    ...
    Chris
    @cmidgley

    Sorry if I was unclear - xsuse works fine, and that's how I figured out the numbers. The documentation, TypeScript file and the other example are all wrong (or at least not portable, as they are wrong on ESP32).

    I do see how to find the base of instrumentation to get slots and instrumentation that follows (and I have that working). I found that before that, System Free Memory, was also offset wrong... Just wondering how I should approach accessing any of the instrumentation - for example, is the base of instrumentation (1) always correct until a particular index, and then we have to scan backwards from the end to find the rest? How would I find System Free Memory or Timers, for example?

    Chris
    @cmidgley

    @phoddie As you know, I've been working on out-of-memory issues, focused on getting preload working. I got much of Instrumentation working (still don't have all the indexes right, see prior comment) and found I needed to use another WeakMap.

    I'm now crashing in xsMapSet.js with an exception here. Spent the last two days trying to debug why and finally got a small fragment showing the crash - it's due to preload being used on a class that gets put into a WeakMap. See this gist for the files that recreate the failure. This is a blocking issue for me as I can't run without preload but crash with it. Hopefully there is a simple solution, but it's beyond me to figure out.

    Peter Hoddie
    @phoddie
    (Hey, @cmidgley. Busy day for me, so I won't get to this right away. Just acknowledging for now.)
    Chris
    @cmidgley
    Appreciate the ack. Understood on being busy. Want me to put in an issue?
    Peter Hoddie
    @phoddie
    Yes, please do. The example is small, which is great. It looks like it gets tripped up because the fully-inverted weak-map implementation assumes the WeakMap keys are mutable. Here the key is an immutable object in ROM. That object can never be collected (since it lives in ROM) but in principle the WeakMap should be able to contain that. We'll see. (Going back to my busy day now!)
    Lawrence R Kincheloe
    @LokiMetaSmith
    hmm, so I got everything to compile with the latest public branch, (there's a bug introduced in xs.h where the riscv compiler throws the size_t define for a loop)
    I think where life is giving me lemons is I'm not able to connect to the dev board over serial, so I can get some proper debug status. I suspect there's an issue in the sdkconfig.defaults
    Peter Hoddie
    @phoddie

    there's a bug introduced in xs.h where the riscv compiler throws the size_t define for a loop

    Is size_t already defined in the riscv build? That change is working on ESP32, ESP32-S2, ESP32-S3, and ESP8266.

    Lawrence R Kincheloe
    @LokiMetaSmith

    yes,

    In file included from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/sys/reent.h:14,
    from /home/loki/Projects/esp-idf/components/newlib/platform_include/sys/reent.h:17,
    from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/sys/errno.h:11,
    from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/errno.h:9,
    from /home/loki/Projects/esp-idf/components/newlib/platform_include/errno.h:18,
    from /home/loki/Projects/moddable/xs/includes/xs.h:130,
    from /home/loki/Projects/moddable/xs/includes/xsmc.h:41,
    from /home/loki/Projects/moddable/modules/pins/digital/digital.c:21:
    /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/lib/gcc/riscv32-esp-elf/8.4.0/include/stddef.h:216:23: error: conflicting types for 'size_t'
    typedef SIZE_TYPE size_t;
    ^~
    In file included from /home/loki/Projects/moddable/xs/includes/xsmc.h:41,
    from /home/loki/Projects/moddable/modules/pins/digital/digital.c:21:
    /home/loki/Projects/moddable/xs/includes/xs.h:113:21: note: previous declaration of 'size_t' was here
    typedef uint32_t size_t;
    ^~
    make: * [/home/loki/Projects/moddable/build/tmp/esp32/esp32c3/debug/helloworld/makefile:760: /home/loki/Projects/moddable/build/tmp/esp32/esp32c3/debug/helloworld/digital.c.o] Error 1

    I just commented out

    // #if defined(ets)
    // typedef uint32_t size_t;
    // #endif

    I'm not sure where "ets" is defined
    '''// #if defined(ets)
    // typedef uint32_t size_t;
    // #endif'''
    Peter Hoddie
    @phoddie
    Sure, that will keep you moving. But it isn't a solution that can be merged since it will break other targets.
    Lawrence R Kincheloe
    @LokiMetaSmith
    hmm, my gitter formatting is not working, but I think you get the idea
    Peter Hoddie
    @phoddie
    __ets__ is defined in the make file for ESP32. It should not be removed as there is code that depends on it.
    Is there a way to check for RISC-V builds of ESP-IDF? That would be way to add a check here to cleanly eliminate the error.
    Lawrence R Kincheloe
    @LokiMetaSmith
    hmm, ok let me step back. usually size_t is defined in the standard library, I'm not sure why its being redefined in xs.h. it looks like that line of code was recently added as well.