## Where communities thrive

• Join over 1.5M+ people
• Join over 100K+ communities
• Free without limits
• Create your own community
##### Activity
• 12:15
kayhayen commented #491
• 12:15
kayhayen commented #491
• 11:54
kayhayen commented #491
• 11:54
kayhayen commented #491
• 11:46
kayhayen commented #510
• 11:46
kayhayen commented #510
• 11:44

kayhayen on factory

Tests: Updates to the CPython t… (compare)

• 11:44

kayhayen on factory

Tests: Updates to the CPython t… (compare)

• 11:41
kayhayen closed #519
• 11:41
kayhayen closed #519
• 11:41
kayhayen commented #519
• 11:41
kayhayen commented #519
• 11:37
kayhayen commented #375
• 11:37
kayhayen commented #375
• 11:36

kayhayen on factory

Eclipse: Remove PyDev annotatio… Windows: Fix, make sure to prov… Tests: Updates to the CPython t… (compare)

• 11:36

kayhayen on factory

Eclipse: Remove PyDev annotatio… Windows: Fix, make sure to prov… Tests: Updates to the CPython t… (compare)

• 11:32
kayhayen labeled #375
• 11:32
kayhayen labeled #375
• 02:58
jayvdb commented #490
• 02:58
jayvdb commented #490
Kay Hayen
@kayhayen
Python3.7 has some progress, but it still gets KeyError somewhere.
Kay Hayen
@kayhayen
so I figured out what to do, and it takes a bit of cleanups
I am refactoring all the places where the module is looked after in sys.modules into proper value returns, and split the top level entry point code off from module entry points locally, so they can have a different interface.
That is a bit tedious, but I am very nearly there I think, but it's tricky of course. The trick is that I pass the intended name via a variable also, and use that. Affects a bit of code right there.
Kay Hayen
@kayhayen
So I think I am there, running internal test suites before pushing this out, quite a bit of cleanup there, but it should only be better, and potentially fixed some other bugs too.
MDTech-us-MAN
@MDTech-us-MAN
hmm... interesting error here:
scons: done building targets.
Traceback (most recent call last):
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\__main__.py", line 184
, in <module>
main()
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\__main__.py", line 177
, in main
MainControl.main()
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\MainControl.py", line
781, in main
executePostProcessing(getResultFullpath(main_module))
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\PostProcessing.py", li
ne 63, in executePostProcessing
lang_id=0,
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\utils\WindowsResources
.py", line 254, in addResourceToFile
update_handle = _openFileWindowsResources(target_filename)
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\utils\WindowsResources
.py", line 161, in _openFileWindowsResources
raise ctypes.WinError()
OSError: [WinError 8] Not enough memory resources are available to process this command
even though the system has plenty of memory
MDTech-us-MAN
@MDTech-us-MAN
I managed to work around it by commenting out executePostProcessing in MainControl and instead writing out the Consts blob to a file
with open("blobs.bin", "wb") as f:
f.write(ConstantCodes.stream_data.stream_data)
then in a separate python session I read in the blobs.bin into ConstantCodes.stream_data.stream_data then run executePostProcessing
after that everything completes successfully and results in a working binary
is there any reason it might not work in the main script
MDTech-us-MAN
@MDTech-us-MAN
I'll test out the importing fix later today
Kay Hayen
@kayhayen
Not enough memory, ouch, never seen that so far.
How much is the memory blob? You know, Nuitka tries to guard against compile time computation of too large constant values, but there might be a gap there, and this could make it include some crazy constant making it huge, how large a file were we talking of @MDTech-us-MAN ?
@JorjMcKie I happily merged jorj and made it part of the pre-release, to which I rebased your branch, good stuff
And the pre-release is out, with substantial more optimization stuff in there, and some important compatibility fixes, I plan on making it a release over the weekend, in fact, the one thing I mean to do right now is to remove the CPython test suite submodules, as they can cause errors in git clones of non-github users, and they probably have no point to most people.
MDTech-us-MAN
@MDTech-us-MAN
The resulting module is 145 MB and the blobs.bin that got written is 3 MB
In fact, I just added subprocess.run([sys.executable, "AddResources.py", getResultFullpath(main_module)])to MainControl after writing the blobs, where AddResources.py simply loads the blobs and runs executePostProcessing
this starts a whole new interpreter and it works perfectly
so system memory isn't an issue
Kay Hayen
@kayhayen
3mb is not that bad though.
MDTech-us-MAN
@MDTech-us-MAN
which makes it all the more weird
let me run it on factory and see what happens
Kay Hayen
@kayhayen
develop is updated just now
I don't think anything changed in that domain, what arch is this, 32 or 64?
MDTech-us-MAN
@MDTech-us-MAN
32
Kay Hayen
@kayhayen
Ok, let me check something.
Nah, the API calls are good. I have to assume, it's Nuitka holding too much RAM in your case really. Which is bad. Can you use --show-memory and see if there is any module that stands out?
MDTech-us-MAN
@MDTech-us-MAN
okay. its running
MDTech-us-MAN
@MDTech-us-MAN
Nuitka:INFO:Total memory usage before generating C code: 1.21 GB (1299304448 bytes):
MDTech-us-MAN
@MDTech-us-MAN
huh, crashed with a memory error on the code generation phase
Traceback (most recent call last):
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\__main__.py", line 184, in <module>
main()
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\__main__.py", line 177, in main
MainControl.main()
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\MainControl.py", line 776, in main
result, options = compileTree(main_module=main_module)
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\MainControl.py", line 669, in compileTree
makeSourceDirectory(main_module=main_module)
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\MainControl.py", line 389, in makeSourceDirectory
module_name=module.getFullName(),
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\codegen\CodeGeneration.py", line 463, in prepareModuleCode
context=context,
File "C:\Panda3D-1.10.5\python\lib\site-packages\nuitka\codegen\ModuleCodes.py", line 100, in getModuleValues
function_body_codes = "\n\n".join(function_body_codes)
MemoryError
got up to over 1.7GB
MDTech-us-MAN
@MDTech-us-MAN
looking through the output, I can't seem to localize a specific module that memory increases on
MDTech-us-MAN
@MDTech-us-MAN
I'm trying it again without show-memory. I'll check the results tomorrow morning. Its getting awfully late here.
Kay Hayen
@kayhayen
Ok, I guess, you need to somehow make this reproducible to me, and I need to check if bytecode demotion is used enough for you, and if it actually succeeds in releasing the memory. Can you raise an issue with an example program or a git clone to make and the command you are using, then I will look into it.
Panda is not small in dependencies, and I want to be sure that it's nothing structurally, e.g. a danging reference to a module somewhere, or a cyclic dependency not caught.
MDTech-us-MAN
@MDTech-us-MAN
I'm not using any dependencies (no --follow-imports or anything similar). All this is just my code.
Paul m. p. P.
@pmp-p
@MDTech-us-MAN: are you trying to compile all the Panda3D modules ? be warned that this is huge and lots of functions use duck typing
MDTech-us-MAN
@MDTech-us-MAN
of course not
like I said. I am only compiling my code right now
my parameters are --module myproject --include-package=myproject
honestly if I'll be able to compile the project packages individually there will be no issue, so let me check if that works
MDTech-us-MAN
@MDTech-us-MAN
And by the way, @kayhayen the overnight build without show-memory completed successfully
MDTech-us-MAN
@MDTech-us-MAN
Also compiling the packages separately works perfectly now! Thank you very much for the quick fix!
Kay Hayen
@kayhayen
Yeah, good improvement, even if I think it ought to not be necessary.
Kay Hayen
@kayhayen
@JorjMcKie good stuff, i have rebased develop due to an issue, and had a small merge conflict with you, please git pull --rebase
btw, are you using the commit hook still? One of the imports seemed unsorted.