Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Peter Hoddie
    @phoddie
    The timer code could sort the list in order of expire time. That's more bookkeeping to get right though.
    Chris
    @cmidgley
    I use a simple guard - set the timer ID to undefined when it expires (before doing the callback) and also when clear is called.
    (and check for undefined though I believe that's not necessary w/clear)
    Peter Hoddie
    @phoddie
    RIght, I always do it that way. Timer.clear is safe to call with undefined. It will just ignore the call silently. Passing it an invalid timer (one that has been cleared, for example), will throw (because that seems like a bad mistake).
    Chris
    @cmidgley

    Very odd. I'm 99.99% confident I am clean on Timer usage. I have only two methods that ever call Timer, they guard it, and I track a unique ID with each usage and trace it. I see exactly what I would expect - several timers created, one expires and when I try to clear a different (not yet expired, long delay) timer it fails with xsGetHostData: invalid. HOWEVER ... this only happens when I proxy the class that contains the Timer, not the Timer itself. I've verified this looks fine in the debugger prior to the failure (it has the ID as I'd expect, for example). I've tried to reproduce this in a small fragment without success. Does anything pop into your head as to why the xsGetHostData error might happen when using a proxy to hold the Timer object?

    I'm looking now into adding decorators so I can mark that class as being disallowed from mocking (I hacked that in and it works) ... but it sure feels like a bug - likely in my code or perhaps something to deal with Host usage in Timer and Proxies? Crazy stuff, sorry. If you have no ideas given how abstract this is, no worries - but I'm burning a lot of hours so I thought I'd ask.

    Peter Hoddie
    @phoddie
    That error happens when you pass an object to Timer.clear that is not a timer object (a timer object which has been cleared -- which includes a one shot timer that has fired -- is considered "not a timer object"). So... when you hit that break, take a look at the argument to Timer.clear on the stack to see what it is.
    Chris
    @cmidgley

    It's a Timer object - as best I can tell, as it says it is "(host) C data" which is what a Timer looks like otherwise.

    I have just gotten it to reproduce! When it's wrapped in a proxy it does this.

    Chris
    @cmidgley
    I would gist it, but gist.github.com appears down. The following reproduces the failure - likely could reduce the proxy logic some but it hopefully will reproduce the failure (and not due to another stupid mistake on my part!)
    import Timer from 'timer';
    
    class MyClass {
        myTimer;
    }
    
    const handler = {
        construct(target, argArray, newTarget) {
            return new Proxy(Reflect.construct(target, argArray, newTarget), handler);
        },
        get(target, propertyKey, receiver) {
            const value = Reflect.get(target, propertyKey, receiver);
            if (propertyKey == 'prototype') return value;
            const proxy = new Proxy(value, handler);
            return proxy;
        },
    };
    
    // choose one of the following sections - the first fails with Proxy, the second works without it
    const ProxyMyClass = new Proxy(MyClass, handler);
    const myProxyInstance = new ProxyMyClass();
    
    // const myProxyInstance = new MyClass();
    
    myProxyInstance.myTimer = Timer.set(() => {
        myProxyInstance.myTimer = undefined;
        trace('long time expired\n');
    }, 300);
    
    Timer.set(() => {
        trace('short timer expired\n');
        if (myProxyInstance.myTimer) Timer.clear(myProxyInstance.myTimer);
    }, 100);
    
    Timer.set(() => {
        trace('done\n');
    }, 2000);
    (and now gist is back up, oh well)
    Peter Hoddie
    @phoddie
    You are proxying the host object!
    Chris
    @cmidgley
    What do you mean by "host object"? The class MyClass?
    Peter Hoddie
    @phoddie
    No. The timer. The handler.get function wraps every returned value in a proxy, which includes myTimer. That won't work, since there's no proxy trap (Reflect.*) for host data.
    Chris
    @cmidgley

    I see - I didn't know that about host data (and therefore Timer). I misunderstood from earlier that I could wrap host functions and didn't realize that host data was different!

    Any way to detect that condition inside the proxy handler?

    Peter Hoddie
    @phoddie
    You end up passing a Proxy instance to Timer.clear. That fails because it isn't a Timer.
    You can wrap host functions but the object used to retrieve the host data needs to be a host object.
    Chris
    @cmidgley
    OK. I do recursive proxy (optionally) and that has no understanding of what it is doing a proxy on. I'm looking now into possibly using typescript decorators.
    Peter Hoddie
    @phoddie
    I understand this code is an excerpt so the big picture is lost. I can't really imagine how to design a solution because I don't understand what this is trying to do.
    Chris
    @cmidgley

    Imagine you have a class with many objects inside it. You want to mock it, so you use a generic mocking service to do it:NewClass = mock(OldClass) (or newObject = mock(new OldClass()). That class that wraps the initial proxy, and as the handler executes it continues to proxy what it finds (get/construct/access).

    No worries - I'll find a way around this. My issue - not Moddable's. Thanks for the help.

    Peter Hoddie
    @phoddie
    The problem is triggered because Timer doesn't call methods of the object, but has static methods (e.g. you call Timer.clear(t) instead of t.clear()). The latter will work through a proxy because the receiver can/will be set correctly.
    If Timer is all that gets in the way (it is a rare API style in the Moddable SDK, mostly to be closer to the web timeout API). you could just make a proxy-friendly implementation of the Timer class.
    Chris
    @cmidgley
    I actually have that, but I need to stop the proxy for recursively doing the Timer itself. Perhaps decorators, perhaps something else. I proved it works though by checking for any property name that contains "timer" and that got my code to work - so now it's just a matter of style and a bit of elbow grease.
    Peter Hoddie
    @phoddie
    You could take inspiration from unscopables and have an unproxyable property to list the property names that shouldn't be proxied. (But even mentioning unscopables in a serious conversation may be grounds for being banished from TC39, so....)
    Chris
    @cmidgley
    I was thinking something like @unmockable with a TS decorator, which would then basically do what unscopables does (if I understand it - I just googled it as I hadn't heard of it before ... but attach a symbol)
    Peter Hoddie
    @phoddie
    With TypeScript + Proxy + Decorators, you are in territory I dare not tread. Too much magic.
    Chris
    @cmidgley
    I respect that. At least this is only used in the test framework, never in the actual product.
    Chris
    @cmidgley
    OK - this is pretty cool... adding a TS decorator to a property executes during preload! Sweet...
    Peter Hoddie
    @phoddie
    Magic. ;)
    Peter Hoddie
    @phoddie
    Coming back around to proxy with host functions, from your example yesterday with Array.prototype.push. Internally to XS there are actually two kinds of host functions – a host object and a host primitive. The host primitive is more compact and less standard. As part of preparing the ROM image, the XS linker converts host objects to host primitives. That step is skipped when strip is disabled (which is why turning off strip allows your Proxy to work). In most places, XS automatically and invisibly promotes a host primitive to a host object when needed (fxToInstance). The special case to do that promotion isn't in the Proxy constructor, so it fails. We'll add that. Just FYI, a more focused workaround is to use Object.assign which promotes the host primitive to a host object: new Proxy(Object.assign([].push), {}).
    Chris
    @cmidgley

    Thanks! Since I can't tell which objects are host, I'd have to do that on every object, property, etc., which I'd prefer not to do. But it sounds like you found the fix?

    I have @unmockable working now and it's resolved my issue with Timer - so I'm all good there! Getting close to have preload working on all modules (one big module left to go). Dependency injection and Mocking are working now (major rework to Mock, a bit of refactoring to Dependency Injection). Hopefully a few more days and I'll see what the new slot footprint looks like.

    Peter Hoddie
    @phoddie
    Yes, we know how to fix it. We'll get that in place. Use whatever workaround you like until then. ;)
    Congrats on the progress of your preload and mock harmonization initiative.
    Chris
    @cmidgley

    Mind if I ask a couple performance/memory questions?

    1) If I have a method that needs to return an object, is there much benefit in slots/memory/fragmentation if I return a native object (return { prop: value }) vs. use a class to contain the object (return newClass(value); where the class provides the prop property)?

    2) Is it correct that if I have an object that is not preloaded with six properties, it will consume 7 slots (1 for the object, and 6 for the properties, or a total of 112 bytes)?

    3) Likewise, if I have an array of 100 elements, it takes 100 slots? (And if the array has objects, we multiple the above with this ... or in this case, 100 elements * 6 slots = 600 slots = 9.6K?)

    4) Would using CDV to encapsulate heavy usage objects (arrays of objects with many properties) drop the slot count to one per CDV buffer?

    Peter Hoddie
    @phoddie
    First, there's no fragmentation. One of the nifty features of the XS allocator and the host implementation on microcontrollers is that neither the slot nor chunk heaps fragment. That's because the slot heap uses a single fixed-size allocation and the chunk heap can be compacted to coalesce free space.
    The most precise answers to your questions can be had by adapting the obscure (and incredibly useful) xsuse example.
    Assigning a value to a property to an object requires a slot. Some values are contained in the property (Boolean, number, null, undefined) others use additional memory (strings use a chunk, for example). The assignment references the existing value in that case, so there's no duplication / cloning.
    compileDataView is great for reducing memory used by the values it can carry. The cost, of course, is they are slower to access - though for many cases that just doesn't matter.
    Peter Hoddie
    @phoddie
    Short answer: use xsuse to measure.
    Chris
    @cmidgley
    Didn't catch xsuse before ... I'll play with it in a few days. Thx.
    Lawrence R Kincheloe
    @LokiMetaSmith

    just pulled the public branch and did the "make install" for linux, got hit with the following errors.
    make install -f simulator.mk
    make[1]: Entering directory '/home/loki/Projects/moddable/build/makefiles/lin'

    simulator debug : cc main.c

    /home/loki/Projects/moddable/build/simulator/lin/main.c: In function ‘onApplicationOpen’:
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:99: warning: ‘.xsa'’ directive output may be truncated writing 5 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:49: note: ‘snprintf’ output between 21 and 8211 bytes into a destination of size 8208
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~~~~~~~~~~~~~~~~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:99: warning: ‘.so'’ directive output may be truncated writing 4 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:49: note: ‘snprintf’ output between 20 and 8210 bytes into a destination of size 8208
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^
    ~~~~~~~~~~~~~~~~~~
    Assembler messages:
    Fatal error: can't create /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o: No such file or directory
    make[1]: [simulator.mk:59: /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o] Error 1
    make[1]: Leaving directory '/home/loki/Projects/moddable/build/makefiles/lin'
    make:
    [makefile:66: install] Error 2

    Michael Kellner
    @mkellner
    @LokiMetaSmith, please "make" before "make install -f simulator.mk"
    Peter Hoddie
    @phoddie

    @cmidgley – a fix the issue you ran into, being unable to create a Proxy for a host function primitive, will be pushed in a couple days. Meanwhile, if you want to give it a try before that, in xsProxy.c after line 780 add these lines:

    #ifdef mxHostFunctionPrimitive
        if ((mxArgc > 0) && (mxArgv(0)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(0));
        if ((mxArgc > 1) && (mxArgv(1)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(1));
    #endif

    There's a corresponding change for Proxy.revocable, but it doesn't sound like you are using that (yet!).

    Chris
    @cmidgley
    Thanks. I don't yet see a need for revocable, but good to know.
    Peter Hoddie
    @phoddie
    Just a matter of time. ;)
    Chris
    @cmidgley
    Feels that way!
    Lawrence R Kincheloe
    @LokiMetaSmith

    @LokiMetaSmith, please "make" before "make install -f simulator.mk"

    yep, that's my issue!

    Chris
    @cmidgley

    Using Instrumentation to get live info on slots - I'm finding the doc and the first example are wrong, as the numbers are not correct (for example, doc says 11 is SlotHeapSize yet it is actually 14 on ESP32). I see in xsuse it scans to find the correct number - that's how I got to 14. What's the story here?

    Also, the type definition for it is very difficult to use. The type InstrumentationOffset is not exported, and really only limits a numeric value and enforces it to be 1-16 (even though different numbers are used). And you can't use the actual names as they are only types and not exported constants. Same issue with WiFi (and perhaps others) on their type definitions. Just an FYI for now, but I think the fix (if instrumentation numbers are static and don't need to be scanned) is to export constants from the .js file and declare them in the .d.ts file (perhaps using a enum, not sure yet). If instrumentation numbers are dynamic, I've got no good ideas for typing ... maybe a dictionary lookup method in .js that does the scan and return the correct number for the instrumentation?

    Peter Hoddie
    @phoddie
    The offsets are different for each port. xsuse was working correctly on ESP32, ESP8266, and macOS very recently. Can take another look.
    Peter Hoddie
    @phoddie
    Just ran xsuse on macOS, ESP8266, and ESP32. All results look reasonable. Here's the start of output from ESP32:
    Boolean: 0 bytes
    Number: 0 bytes
    String `String in ROM`: 0 bytes
    String `fromCharCode(32)`: 12 chunk bytes = 12 bytes
    String `String in ` + 'RAM': 24 chunk bytes = 24 bytes
    Object {}: 1 slots = 16 bytes
    Object {x: 1}: 2 slots = 32 bytes
    Object {x: 1, y: 2}: 3 slots = 48 bytes
    Object {x: 1, y: 2, z: 3}: 4 slots = 64 bytes
    Date: 2 slots = 32 bytes
    BigInt 1n: 12 chunk bytes = 12 bytes
    BigInt 100000000001n: 16 chunk bytes = 16 bytes
    Function () => {}: 7 slots = 112 bytes
    ...
    Chris
    @cmidgley

    Sorry if I was unclear - xsuse works fine, and that's how I figured out the numbers. The documentation, TypeScript file and the other example are all wrong (or at least not portable, as they are wrong on ESP32).

    I do see how to find the base of instrumentation to get slots and instrumentation that follows (and I have that working). I found that before that, System Free Memory, was also offset wrong... Just wondering how I should approach accessing any of the instrumentation - for example, is the base of instrumentation (1) always correct until a particular index, and then we have to scan backwards from the end to find the rest? How would I find System Free Memory or Timers, for example?

    Chris
    @cmidgley

    @phoddie As you know, I've been working on out-of-memory issues, focused on getting preload working. I got much of Instrumentation working (still don't have all the indexes right, see prior comment) and found I needed to use another WeakMap.

    I'm now crashing in xsMapSet.js with an exception here. Spent the last two days trying to debug why and finally got a small fragment showing the crash - it's due to preload being used on a class that gets put into a WeakMap. See this gist for the files that recreate the failure. This is a blocking issue for me as I can't run without preload but crash with it. Hopefully there is a simple solution, but it's beyond me to figure out.