Mind if I ask a couple performance/memory questions?
1) If I have a method that needs to return an object, is there much benefit in slots/memory/fragmentation if I return a native object (return { prop: value }
) vs. use a class to contain the object (return newClass(value);
where the class provides the prop
property)?
2) Is it correct that if I have an object that is not preloaded with six properties, it will consume 7 slots (1 for the object, and 6 for the properties, or a total of 112 bytes)?
3) Likewise, if I have an array of 100 elements, it takes 100 slots? (And if the array has objects, we multiple the above with this ... or in this case, 100 elements * 6 slots = 600 slots = 9.6K?)
4) Would using CDV to encapsulate heavy usage objects (arrays of objects with many properties) drop the slot count to one per CDV buffer?
xsuse
example.
just pulled the public branch and did the "make install" for linux, got hit with the following errors.
make install -f simulator.mk
make[1]: Entering directory '/home/loki/Projects/moddable/build/makefiles/lin'
/home/loki/Projects/moddable/build/simulator/lin/main.c: In function ‘onApplicationOpen’:
/home/loki/Projects/moddable/build/simulator/lin/main.c:434:99: warning: ‘.xsa'’ directive output may be truncated writing 5 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
| ^~
/home/loki/Projects/moddable/build/simulator/lin/main.c:434:49: note: ‘snprintf’ output between 21 and 8211 bytes into a destination of size 8208
434 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.xsa\'", path, gxConfigPath);
| ^~~~~~~~~~~~~~~~~~
/home/loki/Projects/moddable/build/simulator/lin/main.c:427:99: warning: ‘.so'’ directive output may be truncated writing 4 bytes into a region of size between 3 and 8193 [-Wformat-truncation=]
427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
| ^~~
/home/loki/Projects/moddable/build/simulator/lin/main.c:427:49: note: ‘snprintf’ output between 20 and 8210 bytes into a destination of size 8208
427 | snprintf(cmd, sizeof(cmd), "/bin/cp -p \'%s\' \'%s.so\'", path, gxConfigPath);
| ^~~~~~~~~~~~~~~~~~~
Assembler messages:
Fatal error: can't create /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o: No such file or directory
make[1]: [simulator.mk:59: /home/loki/Projects/moddable/build/tmp/lin/debug/simulator/main.o] Error 1
make[1]: Leaving directory '/home/loki/Projects/moddable/build/makefiles/lin'
make: [makefile:66: install] Error 2
@cmidgley – a fix the issue you ran into, being unable to create a Proxy
for a host function primitive, will be pushed in a couple days. Meanwhile, if you want to give it a try before that, in xsProxy.c after line 780 add these lines:
#ifdef mxHostFunctionPrimitive
if ((mxArgc > 0) && (mxArgv(0)->kind == XS_HOST_FUNCTION_KIND))
fxToInstance(the, mxArgv(0));
if ((mxArgc > 1) && (mxArgv(1)->kind == XS_HOST_FUNCTION_KIND))
fxToInstance(the, mxArgv(1));
#endif
There's a corresponding change for Proxy.revocable
, but it doesn't sound like you are using that (yet!).
Using Instrumentation
to get live info on slots - I'm finding the doc and the first example are wrong, as the numbers are not correct (for example, doc says 11 is SlotHeapSize
yet it is actually 14 on ESP32). I see in xsuse
it scans to find the correct number - that's how I got to 14. What's the story here?
Also, the type definition for it is very difficult to use. The type InstrumentationOffset
is not exported, and really only limits a numeric value and enforces it to be 1-16 (even though different numbers are used). And you can't use the actual names as they are only types and not exported constants. Same issue with WiFi
(and perhaps others) on their type definitions. Just an FYI for now, but I think the fix (if instrumentation numbers are static and don't need to be scanned) is to export constants from the .js
file and declare them in the .d.ts
file (perhaps using a enum, not sure yet). If instrumentation numbers are dynamic, I've got no good ideas for typing ... maybe a dictionary lookup method in .js
that does the scan and return the correct number for the instrumentation?
xsuse
on macOS, ESP8266, and ESP32. All results look reasonable. Here's the start of output from ESP32:Boolean: 0 bytes
Number: 0 bytes
String `String in ROM`: 0 bytes
String `fromCharCode(32)`: 12 chunk bytes = 12 bytes
String `String in ` + 'RAM': 24 chunk bytes = 24 bytes
Object {}: 1 slots = 16 bytes
Object {x: 1}: 2 slots = 32 bytes
Object {x: 1, y: 2}: 3 slots = 48 bytes
Object {x: 1, y: 2, z: 3}: 4 slots = 64 bytes
Date: 2 slots = 32 bytes
BigInt 1n: 12 chunk bytes = 12 bytes
BigInt 100000000001n: 16 chunk bytes = 16 bytes
Function () => {}: 7 slots = 112 bytes
...
Sorry if I was unclear - xsuse
works fine, and that's how I figured out the numbers. The documentation, TypeScript file and the other example are all wrong (or at least not portable, as they are wrong on ESP32).
I do see how to find the base of instrumentation to get slots and instrumentation that follows (and I have that working). I found that before that, System Free Memory, was also offset wrong... Just wondering how I should approach accessing any of the instrumentation - for example, is the base of instrumentation (1) always correct until a particular index, and then we have to scan backwards from the end to find the rest? How would I find System Free Memory or Timers, for example?
@phoddie As you know, I've been working on out-of-memory issues, focused on getting preload working. I got much of Instrumentation
working (still don't have all the indexes right, see prior comment) and found I needed to use another WeakMap
.
I'm now crashing in xsMapSet.js
with an exception here. Spent the last two days trying to debug why and finally got a small fragment showing the crash - it's due to preload being used on a class that gets put into a WeakMap
. See this gist for the files that recreate the failure. This is a blocking issue for me as I can't run without preload but crash with it. Hopefully there is a simple solution, but it's beyond me to figure out.
yes,
In file included from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/sys/reent.h:14,
from /home/loki/Projects/esp-idf/components/newlib/platform_include/sys/reent.h:17,
from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/sys/errno.h:11,
from /home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/riscv32-esp-elf/sys-include/errno.h:9,
from /home/loki/Projects/esp-idf/components/newlib/platform_include/errno.h:18,
from /home/loki/Projects/moddable/xs/includes/xs.h:130,
from /home/loki/Projects/moddable/xs/includes/xsmc.h:41,
from /home/loki/Projects/moddable/modules/pins/digital/digital.c:21:
/home/loki/.espressif/tools/riscv32-esp-elf/esp-2021r2-8.4.0/riscv32-esp-elf/lib/gcc/riscv32-esp-elf/8.4.0/include/stddef.h:216:23: error: conflicting types for 'size_t'
typedef SIZE_TYPE size_t;
^~
In file included from /home/loki/Projects/moddable/xs/includes/xsmc.h:41,
from /home/loki/Projects/moddable/modules/pins/digital/digital.c:21:
/home/loki/Projects/moddable/xs/includes/xs.h:113:21: note: previous declaration of 'size_t' was here
typedef uint32_t size_t;
^~
make: * [/home/loki/Projects/moddable/build/tmp/esp32/esp32c3/debug/helloworld/makefile:760: /home/loki/Projects/moddable/build/tmp/esp32/esp32c3/debug/helloworld/digital.c.o] Error 1
I just commented out
// #if defined(ets)
// typedef uint32_t size_t;
// #endif
#if defined(__ets__) && !defined(ESP32)
won't define it on ESP32 targets. Would that work on the RISC-V target?
Once again a crazy question.... Is there any way to cause code executed during preload to execute after all modules have been preloaded?
Background: I have unit test files (Jest-like) that if I include in the Manifest they are detected and executed. I used to use Modules
to find them, but now I am preloading them - which is really cool because all sorts of overhead gets absorbed into flash. As each module is preloaded, when the Jest describe
method is execute, I register the test suite. Once all modules are preloaded, I'd like to freeze that object - but I can't do it in main
(it runs before they do) and I don't see a way to cause code to execute after they have been loaded. There are ways around this (I could have a module that is just for loading tests), but that has the side effect that you have to keep it and the manifest synchronized or it will fail to work.
Also - can you confirm that slot/chunk allocation used during preload, even if they garbage collect out, will keep the "high water" mark of the slot and chunk usage when the code executes? What I think I see happening is after a big preload (the test stuff above) my slot and chunk numbers are decent (say 8K and 2K), but the allocated size is 80K and 20K. The problem is that when the code runs, it needs less slots and more chunk space available, which garbage collection won't help with.