Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 09:44
    belonesox edited #1139
  • 09:44
    belonesox edited #1139
  • 07:03
    kayhayen labeled #1139
  • 07:03
    kayhayen labeled #1139
  • 07:03
    kayhayen assigned #1139
  • 07:03
    kayhayen assigned #1139
  • 07:03
    kayhayen commented #1139
  • 07:03
    kayhayen commented #1139
  • 07:00
    kayhayen commented #1137
  • 07:00
    kayhayen commented #1137
  • 06:57
    kayhayen commented #799
  • 06:57
    kayhayen commented #799
  • 00:00
    duolabmeng6 commented #799
  • 00:00
    duolabmeng6 commented #799
  • Jun 21 21:17
    belonesox opened #1139
  • Jun 21 21:17
    belonesox opened #1139
  • Jun 21 16:45
    Hintay commented #1137
  • Jun 21 16:45
    Hintay commented #1137
  • Jun 21 14:58
    ryancinsight commented #1127
  • Jun 21 14:58
    ryancinsight commented #1127
Kay Hayen
@kayhayen
There is an important regression of 0.6.15 related to generic aliases, a 3.9 feature, that got optimization in Nuitka, but apparently can lead to crashes, the fix is going to be in 0.6.15.2 and is on that branch right now and factory. I plan a hotfix for today.
Felix Ingram
@lllama

Hello all. I'm doing multiple builds of my app. I change a single file between builds to update some constants and then compile again. I use a different --output-dir for each build and I'm not seeing clcache getting any hits (it states everything misses). Is my changing of the output directory affecting the cache?

(I'm building on a Windows docker image in AWS CodeBuild - it's currently taking over an hour to build four versions, so anything to reduce that would be great)

Felix Ingram
@lllama
Looks like it's the changing output dir. I did two identical builds and it hits the cache. I'm trying a different approach that uses the same output dir - let's see if that gets hits rather than misses
Kay Hayen
@kayhayen
I think clcache which is really Nuitka clcache, is a victim of the MS compiler there.
I think I have seen that there is the build path in object files, and I didn't find a way to get rid of them yet. Neither did I find something in Visual Studio that causes it to be there, nor a way to ignore it in clcache, although it does something. I would need help by experts here @lllama but it seems clcache has that problem. I think paths are becoming absolute.
Nuitka does however work hard to only ever provide relative paths in the scons. Prove me wrong there. But in my book that means, the MS compiler does it. And when hashing things up, yeah.
This is very sad, my CI suffers from having to have much too large caches to benefit.
IF you can preserve the caches however, it will be better.
zygon.sisyphus
@zygon.sisyphus:matrix.org
[m]
hi, is possible to change on windows mingw sdk ? how ? https://github.com/brechtsanders/winlibs_mingw/releases/
Kay Hayen
@kayhayen
With a pull request that modifies the URLs in Nuitka
zygon.sisyphus
@zygon.sisyphus:matrix.org
[m]
ok, thx
Kay Hayen
@kayhayen
I wish I had an idea, how to make this work: --include-data-file=data/**/*.txt=data
The first thing is a pattern that matches several files, and the later a root folder to put them in, but with what directory base.
I guess, I need to make people specify the root, then the pattern, and then the target, like so:
--include-data-file=data=*/.txt=data
I guess, I am going to add support for a new kind of thing then, with '|' as the separator.
--include-data-file=data|data|*/.txt
Except that pipe is not going to work well with bash... and colons work badly for paths
so, just counting the number of "=" then, we could be good.
Kay Hayen
@kayhayen
Yeah... recursive support is working, also thanks to "glob2" module for older Pythons
Kay Hayen
@kayhayen
so... "-o" works with onefile too now and the tensorflow plugin was using forbidden APIs.
I guess, it's not used a lot.
Kay Hayen
@kayhayen
Took a bit of time off, to get FreeBSD to work nicely, and added onefile support for it too.
Kay Hayen
@kayhayen
Anybody here fit with FreeBSD? I would need somebody to get the dtruss based testing to work for the onefile tests.
brev-dev
@brev-dev
How does one try the new "--noinclude-setuptools-mode=nofollow" option? If I just add this to my existing command, I get the error "main.py: error: no such option: --noinclude-setuptools-mode".
Kay Hayen
@kayhayen
You need to enable the plugin that provides it too. In this case, anti-bloat.
Then do --help and checkout the new options, I am continously adding stuff.
There is e.g. also a new treatment coming for psutils that avoids platform specifics that do not apply.
Kay Hayen
@kayhayen
So, --lto is now the default on Windows with MSVC 2019, not sure I want to allow to disable it at all. It's worth 10-20% performance.
And size gain, and might compile faster, not sure.
Kay Hayen
@kayhayen
Even more love for the anti-bloat plugin, it's now a yaml file, to allow for easy extension by you:
Please checkout this, and let me know if you have additions or comments on that:
Kay Hayen
@kayhayen
This could become huge. We need to add stuff there that removes useless code causing bloat. There will be other examples, and quite generally this could become a major tool to fight dependency creep.
And the mechanism could be used by other plugins that current do search and replaces of on their own. But then regular expressions, and callbacks might be needed, so maybe not all will be doable in that fashion.
Kay Hayen
@kayhayen
Better threading for 3.8 is a thing too.
Kay Hayen
@kayhayen
New pre-release with the good stuff, incl. the yaml anti-bloat plugin stuff, 3.8 speedup, Windows MSVC 2019 using LTO by default, and plenty other improvements.
Very close to release this is now. I am just making a few final touches I think over the coming days.
krnr
@krnr
hello Kay and all the guys!
first of all, let me to express my gratitude for developing such a great instrument. I used it to compile simple scripts with great success.
however, i really lack instructions of what to do with bigger applications. e.g. what if you have celery worker processing tasks? how do you run it, import tasks? (probably extracting all logic to modules which are imported in task handlers? but what if it is possible to pack whole celery app?)
what if you have a [A/W]SGI app? e.g. gunicorn relies on my_file:my_app_callable syntax to run apps. how will it work with --standalone?
what if you have a torch module invoked by http call in your wsgi app? I tried to use --plugin-enable=torch and couldn't wait till the build is over ))) it lasted probably an hour when my patience was over.
so, generally speaking, i lack the recipes. where could I head at to find some inspiration?
basically, i believe that vast amount of recipes and snippets made django more popular than other web frameworks (not "battery included"). and now we can see the same with fastapi/pydantic or attrs - good bunch of examples make it more attractive to use and clicks immediately "oh, that's my case". I would love to see the same for nuitka.
Kay Hayen
@kayhayen
@krnr well, I am giving commercial users support if that's what you are looking for. But creating examples, etc. for people to use, I cannot really do that. When people will pay me to solve their deployment, I can work on that, and make sure it's reusable information, but I cannot just do it for free while developing a Python compiler for free, not going to happen. So either you invest the time to figure it out and send me PRs with documentation improvements, or you pay me to do it, or it's not happening.
1 reply
Alex Wen
@qyalexwen
I tried to use nuitka in my shared hosting server, but the admin said "We found that the Nuitka module was generating high resource usage and hitting the account's limit.". The nuitka got an error:

python -m nuitka a.py

....
File "/home/erposcom/venv/py383/lib/python3.8/site-packages/nuitka/build/SconsSpawn.py", line 241, in runSpawnMonitored
thread.start()
File "/home/erposcom/lib/python3.8/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
scons: * [static_src/CompiledFunctionType.o] Cannot allocate memory
Is there a way not to use threading when compiling? Thanks.
Ben Fässler
@Ben-Faessler
Did you try '-j 1' in the command?
Kay Hayen
@kayhayen
@qyalexwen make sure to use ccache, to avoid compiling C at all. And we need to get your caching out @Ben-Faessler sorry for not finishing it
And yes, -j1 might be a very good idea indeed. Also checking develop and the anti-bloat plugin, and make certain you do not include the usual trouble makes, setuptools, pytest, etc. and compiling in a venv is very import too, which you do.
Otherwisse, use WSL 2 if you have a Windows desktop, that should be easier than a remote machine with limited power.
Wu Tiecheng
@MrBox007
@kayhayen Hello, is it possible that nuitka can package python dashborad codes(UI based on browser) like dash, panel, streamlit?
Kay Hayen
@kayhayen
@MrBox007 that's an unknown