Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Peter Hoddie
    @phoddie
    Create a binding for xsCollectGarbage (documented in XS in C. Or just use the debug module, which provides it already.
    Chris
    @cmidgley

    great... how?

    I am now confident there is a bug in Timer ... I think it deals with garbage collection and use of host memory, but was hoping to isolate further before sending your way. I can repo the Timer bug in just a few lines, but I need to jump to debugger and back to do it (that does a GC)

    Chris
    @cmidgley

    Timer fails for me with an xsGetHostData: invalid in my product. I have it recreated as long as I do a debugger statement at the end (I tried various Debug.gc() calls but that didn't seem to exacerbate it), so not sure what else debugger is doing that also occurs in my product (without using debugger) but I'm out of ideas. Here is the fragment that causes the failure (run, hit breakpoint, and continue running to see the exception):

    import Timer from 'timer';
    
    const myTimer = Timer.set(() => {}, 300);
    Timer.set(() => Timer.clear(myTimer), 100);
    debugger;

    Do you want me to make GitHub issues for this and/or the Proxy host function issues?

    Peter Hoddie
    @phoddie
    The thing that debugger is changing is time. The code is working as intended, but I can see how that can be confusing here. I can explain later.
    Chris
    @cmidgley
    No need - I see the silly mistake I made here (clearing myTimer after it expired). Adding a guard on that fixed it for this test (sadly, the same guard was already in my main code ... but I'll figure it out!)
    Peter Hoddie
    @phoddie
    Right. The thing i... you would the second timer (100ms) to execute before the first timer (300ms). But when you stop in the debugger, they both end up being eligible to fire at the same time. The timer implementation doesn't guarantee the order of callbacks in the case, so here Timer.clear ends up executing after the set expires.
    Chris
    @cmidgley
    Yes, totally makes sense and I had that in the main code. I just removed too much in my test! I'm working it...
    Peter Hoddie
    @phoddie
    The timer code could sort the list in order of expire time. That's more bookkeeping to get right though.
    Chris
    @cmidgley
    I use a simple guard - set the timer ID to undefined when it expires (before doing the callback) and also when clear is called.
    (and check for undefined though I believe that's not necessary w/clear)
    Peter Hoddie
    @phoddie
    RIght, I always do it that way. Timer.clear is safe to call with undefined. It will just ignore the call silently. Passing it an invalid timer (one that has been cleared, for example), will throw (because that seems like a bad mistake).
    Chris
    @cmidgley

    Very odd. I'm 99.99% confident I am clean on Timer usage. I have only two methods that ever call Timer, they guard it, and I track a unique ID with each usage and trace it. I see exactly what I would expect - several timers created, one expires and when I try to clear a different (not yet expired, long delay) timer it fails with xsGetHostData: invalid. HOWEVER ... this only happens when I proxy the class that contains the Timer, not the Timer itself. I've verified this looks fine in the debugger prior to the failure (it has the ID as I'd expect, for example). I've tried to reproduce this in a small fragment without success. Does anything pop into your head as to why the xsGetHostData error might happen when using a proxy to hold the Timer object?

    I'm looking now into adding decorators so I can mark that class as being disallowed from mocking (I hacked that in and it works) ... but it sure feels like a bug - likely in my code or perhaps something to deal with Host usage in Timer and Proxies? Crazy stuff, sorry. If you have no ideas given how abstract this is, no worries - but I'm burning a lot of hours so I thought I'd ask.

    Peter Hoddie
    @phoddie
    That error happens when you pass an object to Timer.clear that is not a timer object (a timer object which has been cleared -- which includes a one shot timer that has fired -- is considered "not a timer object"). So... when you hit that break, take a look at the argument to Timer.clear on the stack to see what it is.
    Chris
    @cmidgley

    It's a Timer object - as best I can tell, as it says it is "(host) C data" which is what a Timer looks like otherwise.

    I have just gotten it to reproduce! When it's wrapped in a proxy it does this.

    Chris
    @cmidgley
    I would gist it, but gist.github.com appears down. The following reproduces the failure - likely could reduce the proxy logic some but it hopefully will reproduce the failure (and not due to another stupid mistake on my part!)
    import Timer from 'timer';
    
    class MyClass {
        myTimer;
    }
    
    const handler = {
        construct(target, argArray, newTarget) {
            return new Proxy(Reflect.construct(target, argArray, newTarget), handler);
        },
        get(target, propertyKey, receiver) {
            const value = Reflect.get(target, propertyKey, receiver);
            if (propertyKey == 'prototype') return value;
            const proxy = new Proxy(value, handler);
            return proxy;
        },
    };
    
    // choose one of the following sections - the first fails with Proxy, the second works without it
    const ProxyMyClass = new Proxy(MyClass, handler);
    const myProxyInstance = new ProxyMyClass();
    
    // const myProxyInstance = new MyClass();
    
    myProxyInstance.myTimer = Timer.set(() => {
        myProxyInstance.myTimer = undefined;
        trace('long time expired\n');
    }, 300);
    
    Timer.set(() => {
        trace('short timer expired\n');
        if (myProxyInstance.myTimer) Timer.clear(myProxyInstance.myTimer);
    }, 100);
    
    Timer.set(() => {
        trace('done\n');
    }, 2000);
    (and now gist is back up, oh well)
    Peter Hoddie
    @phoddie
    You are proxying the host object!
    Chris
    @cmidgley
    What do you mean by "host object"? The class MyClass?
    Peter Hoddie
    @phoddie
    No. The timer. The handler.get function wraps every returned value in a proxy, which includes myTimer. That won't work, since there's no proxy trap (Reflect.*) for host data.
    Chris
    @cmidgley

    I see - I didn't know that about host data (and therefore Timer). I misunderstood from earlier that I could wrap host functions and didn't realize that host data was different!

    Any way to detect that condition inside the proxy handler?

    Peter Hoddie
    @phoddie
    You end up passing a Proxy instance to Timer.clear. That fails because it isn't a Timer.
    You can wrap host functions but the object used to retrieve the host data needs to be a host object.
    Chris
    @cmidgley
    OK. I do recursive proxy (optionally) and that has no understanding of what it is doing a proxy on. I'm looking now into possibly using typescript decorators.
    Peter Hoddie
    @phoddie
    I understand this code is an excerpt so the big picture is lost. I can't really imagine how to design a solution because I don't understand what this is trying to do.
    Chris
    @cmidgley

    Imagine you have a class with many objects inside it. You want to mock it, so you use a generic mocking service to do it:NewClass = mock(OldClass) (or newObject = mock(new OldClass()). That class that wraps the initial proxy, and as the handler executes it continues to proxy what it finds (get/construct/access).

    No worries - I'll find a way around this. My issue - not Moddable's. Thanks for the help.

    Peter Hoddie
    @phoddie
    The problem is triggered because Timer doesn't call methods of the object, but has static methods (e.g. you call Timer.clear(t) instead of t.clear()). The latter will work through a proxy because the receiver can/will be set correctly.
    If Timer is all that gets in the way (it is a rare API style in the Moddable SDK, mostly to be closer to the web timeout API). you could just make a proxy-friendly implementation of the Timer class.
    Chris
    @cmidgley
    I actually have that, but I need to stop the proxy for recursively doing the Timer itself. Perhaps decorators, perhaps something else. I proved it works though by checking for any property name that contains "timer" and that got my code to work - so now it's just a matter of style and a bit of elbow grease.
    Peter Hoddie
    @phoddie
    You could take inspiration from unscopables and have an unproxyable property to list the property names that shouldn't be proxied. (But even mentioning unscopables in a serious conversation may be grounds for being banished from TC39, so....)
    Chris
    @cmidgley
    I was thinking something like @unmockable with a TS decorator, which would then basically do what unscopables does (if I understand it - I just googled it as I hadn't heard of it before ... but attach a symbol)
    Peter Hoddie
    @phoddie
    With TypeScript + Proxy + Decorators, you are in territory I dare not tread. Too much magic.
    Chris
    @cmidgley
    I respect that. At least this is only used in the test framework, never in the actual product.
    Chris
    @cmidgley
    OK - this is pretty cool... adding a TS decorator to a property executes during preload! Sweet...
    Peter Hoddie
    @phoddie
    Magic. ;)
    Peter Hoddie
    @phoddie
    Coming back around to proxy with host functions, from your example yesterday with Array.prototype.push. Internally to XS there are actually two kinds of host functions – a host object and a host primitive. The host primitive is more compact and less standard. As part of preparing the ROM image, the XS linker converts host objects to host primitives. That step is skipped when strip is disabled (which is why turning off strip allows your Proxy to work). In most places, XS automatically and invisibly promotes a host primitive to a host object when needed (fxToInstance). The special case to do that promotion isn't in the Proxy constructor, so it fails. We'll add that. Just FYI, a more focused workaround is to use Object.assign which promotes the host primitive to a host object: new Proxy(Object.assign([].push), {}).
    Chris
    @cmidgley

    Thanks! Since I can't tell which objects are host, I'd have to do that on every object, property, etc., which I'd prefer not to do. But it sounds like you found the fix?

    I have @unmockable working now and it's resolved my issue with Timer - so I'm all good there! Getting close to have preload working on all modules (one big module left to go). Dependency injection and Mocking are working now (major rework to Mock, a bit of refactoring to Dependency Injection). Hopefully a few more days and I'll see what the new slot footprint looks like.

    Peter Hoddie
    @phoddie
    Yes, we know how to fix it. We'll get that in place. Use whatever workaround you like until then. ;)
    Congrats on the progress of your preload and mock harmonization initiative.
    Chris
    @cmidgley

    Mind if I ask a couple performance/memory questions?

    1) If I have a method that needs to return an object, is there much benefit in slots/memory/fragmentation if I return a native object (return { prop: value }) vs. use a class to contain the object (return newClass(value); where the class provides the prop property)?

    2) Is it correct that if I have an object that is not preloaded with six properties, it will consume 7 slots (1 for the object, and 6 for the properties, or a total of 112 bytes)?

    3) Likewise, if I have an array of 100 elements, it takes 100 slots? (And if the array has objects, we multiple the above with this ... or in this case, 100 elements * 6 slots = 600 slots = 9.6K?)

    4) Would using CDV to encapsulate heavy usage objects (arrays of objects with many properties) drop the slot count to one per CDV buffer?

    Peter Hoddie
    @phoddie
    First, there's no fragmentation. One of the nifty features of the XS allocator and the host implementation on microcontrollers is that neither the slot nor chunk heaps fragment. That's because the slot heap uses a single fixed-size allocation and the chunk heap can be compacted to coalesce free space.
    The most precise answers to your questions can be had by adapting the obscure (and incredibly useful) xsuse example.
    Assigning a value to a property to an object requires a slot. Some values are contained in the property (Boolean, number, null, undefined) others use additional memory (strings use a chunk, for example). The assignment references the existing value in that case, so there's no duplication / cloning.
    compileDataView is great for reducing memory used by the values it can carry. The cost, of course, is they are slower to access - though for many cases that just doesn't matter.
    Peter Hoddie
    @phoddie
    Short answer: use xsuse to measure.
    Chris
    @cmidgley
    Didn't catch xsuse before ... I'll play with it in a few days. Thx.
    Lawrence R Kincheloe
    @LokiMetaSmith

    just pulled the public branch and did the "make install" for linux, got hit with the following errors.
    make install -f simulator.mk
    make[1]: Entering directory '/home/loki/Projects/moddable/build/makefiles/lin'

    simulator debug : cc main.c

    /home/loki/Projects/moddable/build/simulator/lin/main.c: In function ‘onApplicationOpen’:
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:99: warning: ‘.xsa'’ directive output may be truncated writing 5 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:434:49: note: ‘snprintf’ output between 21 and 8211 bytes into a destination of size 8208
    434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
    | ^~~~~~~~~~~~~~~~~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:99: warning: ‘.so'’ directive output may be truncated writing 4 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^~~
    /home/loki/Projects/moddable/build/simulator/lin/main.c:427:49: note: ‘snprintf’ output between 20 and 8210 bytes into a destination of size 8208
    427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
    | ^
    ~~~~~~~~~~~~~~~~~~
    Assembler messages:
    Fatal error: can't create /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o: No such file or directory
    make[1]: [simulator.mk:59: /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o] Error 1
    make[1]: Leaving directory '/home/loki/Projects/moddable/build/makefiles/lin'
    make:
    [makefile:66: install] Error 2

    Michael Kellner
    @mkellner
    @LokiMetaSmith, please "make" before "make install -f simulator.mk"
    Peter Hoddie
    @phoddie

    @cmidgley – a fix the issue you ran into, being unable to create a Proxy for a host function primitive, will be pushed in a couple days. Meanwhile, if you want to give it a try before that, in xsProxy.c after line 780 add these lines:

    #ifdef mxHostFunctionPrimitive
        if ((mxArgc > 0) && (mxArgv(0)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(0));
        if ((mxArgc > 1) && (mxArgv(1)->kind == XS_HOST_FUNCTION_KIND))
            fxToInstance(the, mxArgv(1));
    #endif

    There's a corresponding change for Proxy.revocable, but it doesn't sound like you are using that (yet!).

    Chris
    @cmidgley
    Thanks. I don't yet see a need for revocable, but good to know.
    Peter Hoddie
    @phoddie
    Just a matter of time. ;)