Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 03 18:18

    kayhayen on factory

    Release: Fix, wasn't detecting … Python3.9: Disable warning give… Changelog updates (compare)

  • Dec 03 18:18

    kayhayen on factory

    Release: Fix, wasn't detecting … Python3.9: Disable warning give… Changelog updates (compare)

  • Dec 03 17:49
    Pro100rus32 commented #888
  • Dec 03 17:49
    Pro100rus32 commented #888
  • Dec 03 16:15
    kayhayen commented #148
  • Dec 03 16:15
    kayhayen commented #148
  • Dec 03 16:08
    kayhayen commented #887
  • Dec 03 16:08
    kayhayen commented #887
  • Dec 03 16:07
    kayhayen labeled #887
  • Dec 03 16:07
    kayhayen labeled #887
  • Dec 03 16:07
    kayhayen unlabeled #887
  • Dec 03 16:07
    kayhayen unlabeled #887
  • Dec 03 16:07
    kayhayen closed #888
  • Dec 03 16:07
    kayhayen closed #888
  • Dec 03 16:07
    kayhayen commented #888
  • Dec 03 16:07
    kayhayen commented #888
  • Dec 03 16:05
    kayhayen assigned #888
  • Dec 03 16:05
    kayhayen assigned #888
  • Dec 03 16:05
    kayhayen labeled #888
  • Dec 03 16:05
    kayhayen labeled #888
Kay Hayen
@kayhayen
I mean gcc and MSVC, but it still shaves off some hundred bytes, so I will keep it.
Kay Hayen
@kayhayen
Does anybody have experience with http://winlibs.com/ ?
It seems they got gcc 10.2, and even include clang for good measure in a zip file.
If that works, Nuitka could download that automatically and use it right away, and get rid of the nasty problems C novices have picking stuff.
Kay Hayen
@kayhayen
Seems to work for me, and it's much easier than the old style MinGW64 downloads.
Very exciting, yet another roadblock will be gone for beginners. Now that ccache and gcc, dependency walker are downloaded, that is going to remove a problem people had in the past.
Stas Fomin
@belonesox

Actually, I cannot get, why Nuitka has two mutually excluded options
— "standalone" for binary compiling with collecting all shared libs, and
module compiling, without collecting shared libs.

Of course, if we can patch here https://github.com/Nuitka/Nuitka/blob/8f794f16117e8d5507336f3d28935f7799f640d7/nuitka/importing/Recursion.py#L180
we can get module compilation with all needed SHLIBs.

So, it seems that module compilation should get all needed shared libs by default,
ot may be some new option like "--recurse-shared-libs" should exists to compile module with all needed shared libs.

My goal — precompile compile fat modules (like cv2/scipy/… ) with all needed shared libs,
and get fast recompilation of my part of program with ("--recurse-not-to=cv2", "--recurse-not-to=scipy")
and even without ('--plugin-enable=numpy'), and just copying precompiled modules staff to
standalone dist.

But may be missed something out and my goal is impossible?

Kay Hayen
@kayhayen
@belonesox modules must be loaded into an existing Python installation that has all the libraries.
therefore they are not standalone by design, they are to plugin to arbitrary Python installations meeting their version checks
I have seen people create .so to replace a lot of Python code, and then use Nuitka standalone compilation to pick that up. May get hard with things that need plugins to e.g. inject code or patch it, but not impossible, but I of course would much rather work on getting caching to work for the Python level compilation.
So I am definitely not in on these attempts, no.
Stas Fomin
@belonesox

modules must be loaded into an existing Python installation that has all the libraries.
Therefore they are not standalone by design, they are to plugin to arbitrary
Python installations meeting their version checks.

Lets consider some samples.
Python "cairo" module consists of python part (__init__.py etc)
and binary part "_cairo.*.so".
These two parts are inseparable.

Of course, there can be another deps even to system libraries outside python distribution
(so there are a lot of modules violated contract "a plugin to arbitrary Python installations meeting their version checks", for example libGDAL deps, etc.), but why not grab all staff definitely needed by a python module?

Nuitka in "--module " mode compiles only python part, and makes completely useless "cairo.*.so".
In "--module" + 'standalone patch' it get "cairo.*.so" + "_cairo.*.so", that should be working.

Why this (grab binary shared libs deps) is not OK?

Is the modules compiled with "--module"
cannot be used (by copying) with standalone projects compiled with "--recurse-not-to=" (these modules, compiled separately)?

If they cannot — then why, is this complex design problem?

Of course, standalone Nuitcompiling all staff with all modules open a lot of optimization possibilities, but reusing precompiled modules will be very useful for speedup development/testing cycle.

AutoBonus
@AutoBonus
So I am trying to compile my script but the issue is.. for someone reason numpy is included in it even though I haven't used it!
and cryptography too!
Kay Hayen
@kayhayen
@AutoBonus that is the Python way, if you import e.g. "pandas" you include just about everything imaginable and unimaginable.
@belonesox for that to work in standalone, we actually have had to do work :)
Kay Hayen
@kayhayen
@belonesox I absolutely agree with the benefits it will have. What you probably didn't get, that is I envision this to happen by compiling e.g. numpy once, storing some XML files with the result. That will take some time, but next time, it only checks the checksum of the numpy module (and the roughly 100? in there), and loads that, super quick.
From that it generates code, that should not take much time, but if it did, we could cache the C files too, no big deal. But there is something about module constants, that need to be allocated, which only happens then, so that would also have to cached. Loading that will be super quick.
Then the C compiler already caches things with ccache pretty good, clcache seems to be bugged by a difference as at least MSVC includes that build directory path.
They do the same with C code. So only final assembly of the executable and collecting the DLLs remains, which is relatively quick depending on your IO capability, but SSD speeds make that not a problem.
In this world, I do not need anything pre-compiled. So am I going to run an implement what you suggested @belonesox or will I spend that time towards a full caching solution? My answer was given. I am not going to waste my time with manual solutions like that.
Kay Hayen
@kayhayen
btw, if compilation is slow, use develop, I spend a lot of time of increasing scalability of both Python run time and C compilation, the next release is going to reduce times by a lot.
And avoid MSVC if you can, at least I believe it's slower to compile, that that might have been 2015 version last I looked.
Emre
@emrecpp
@kayhayen Hi, You said "Use ccache or clcache, these will reuse object files @emrecpp " "As for Python level caching, not yet there, but I hope to have it eventually." Do you Have a date in your mind which one supports level caching so we can compile %50+ faster? Which nuitka version you think to do this? Thanksd
Kay Hayen
@kayhayen
Tommorrow at latest of course @emrecpp
I am not yet working on these, Python 3.9, the 0.6.10 release, and generally issues kept me away from it. Not sure when I will be putting this on my agenda. And I think 50%+ faster would be very conservative. But there was a person on Github that wanted to volunteer. There is something that I did, a locals object cache, that prevents Nuitka from just in-process testing if XML stream and restream works. I was going to add the infrastructure for handling cache files, and this person might take it from there to fix the likely many regressions we have. Getting the XML format to 100% completely save and load restore accurate, that might be a lot of work.
The only good news is that in December I practically have holiday all month. Let's see what gets done in that time. But I would be surprised if caching was in that. Probably more optimized C code generation, finally using special code for lists, much like we now do for dictionary operations.
Emre
@emrecpp

Thanks for answer.

This feature is very important for me. Because I am compiling multiple times a day.

Kay Hayen
@kayhayen
Why are you @emrecpp ? Can't you test your app with pure Python, and only build when you approach release?
Kay Hayen
@kayhayen
I found a matplotlib issue for Debian versions of it, we make assumptions that matplotlib package data resides in the package folder, but they put it into the '/usr/share/matplotlib2/' according to their policy. I added a query of that location to the plugin handling that.
Day jobbing, but there is a bunch of things to come, then I can release this weekend I think.
Emre
@emrecpp
@kayhayen there is always a bug. I need to fix and compile it again.
Kay Hayen
@kayhayen
You mean a bug from you packaging it, or in the code?
Emre
@emrecpp
in the code, but my project has more 30.000 lines so compiling takes a lot time :/
Kay Hayen
@kayhayen
But it might be from dependencies, if you are using standalone mode, that can quickly escalate. However, in some report, I discovered that Nuitka is triggering too many global passes. If that holds true, it will reduce the Nuitka compile time by a lot potentially, because the worst loop in a program and its dependencies determine the amount of global passes.
But you didn't really answer @emrecpp if there is a bug, what is the typical cause, you fiddling with what to include or not, or a Python bug, because well, I am curious to know how you are working. You do not test with the original code then?
Kay Hayen
@kayhayen
Still working on moving away from Anaconda to winlibs on the Github actions and adding 3.9 tests there, which reveal a bunch of error message changes only, that I need to follow, small stuff mostly, but right now, I made a replacement for PyList_Extend that should be much faster itself, and faster to call, and cause less overhead, because it has a bool error indicator instead of a PyObject *, so it needs not be released.
Emre
@emrecpp
Thanks for answer and sorry for my bad English. Example, i compiled my program and changed 1 string in exe, try to compile again it takes 20 minutes in 16thread cpu 16gb ram PC. Here used modules list: https://paste.ofcode.org/3vW9bkNcKY3UWtzRPWSBRW , firebase, pyqt5, sqlite, cryptography, win32api... There is a lot of modules so compile time is increasing. Thanks.
NeuralAIM
@NeuralAIM
Hello! How can I download the previous factory version?
Kay Hayen
@kayhayen
I think I want 3.9 tests to run smooth, as well as winlibs before the release. But this is taking a lot of tries to get right. Starting today I am on holiday for next week and then the next week I will work for only a day. I guess, lots of time to sprint for more optimization. It's a shame that the list operations with know type shape are not optimized, and of course, that do not have a real PyList_Append. But as it's only a subset of PyList_Extend code, it will be easy to add it.
Kay Hayen
@kayhayen
Forgot to press enter on this yesterday, I already have that 3.9 stuff on factory, and it's using the winlibs compiler no problem. Good stuff.
And I think, I mostly finished the changelog work, which is HUGE. Never did we have this many new features, this much optimization, although there seem to be few bug fixes. I need to check with the hotfixes, why we have 7 of them, and so little bugs in the changelog, that seems off, I guess, it's because I compared master with develop branch for commits, but I should have looked at where 0.6.9 tag was.
Kay Hayen
@kayhayen
And now automatic download is operational on develop, also a few Python 3.9 tweaks, enought to have added it to Github actions. In the course, I need to optimize our extend operation as I said, so that's there too now, even if not yet used for list.extend it will be in the future, and it would be necessary for optimal unpacking on 3.9 anyway.
aarushans
@aarushans
Hey guys I'm new to nuitka and I want to convert a .py file to a .exe file. Can someone tell me how to do this?
Kay Hayen
@kayhayen
@aarushans read the user manual, it should get you covered
Kay Hayen
@kayhayen
So develop was a big regressive until today. The replacement for _PyList_Extend wasn't quite bug free for a few rounds.
I rebased develop though, and pushed the correct code now, which also contains the PyList_Append replacement.
Then I looked into Speedcenter, and hopefully fixed its bugs, the caching wasn't working at all anymore, and the stable Nuitka outputs scons stuff even if using --quiet, which I never backported due to it being too risky.
I hope to take a look at the performance numbers and check out if everything is as can be expected, after which I think I can make the release finally.
Hopefully this will be the cleanest for a while. Sure people will complain about --onefile not allowing any placement of extra files, or Windows not compressing as much.