Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 08:10
    kayhayen assigned #622
  • 08:10
    kayhayen assigned #622
  • 08:10
    kayhayen milestoned #622
  • 08:10
    kayhayen milestoned #622
  • 08:09

    kayhayen on develop

    Fix, mac ctypes.util does not h… (compare)

  • 08:09

    kayhayen on develop

    Fix, mac ctypes.util does not h… (compare)

  • 08:09
    kayhayen closed #622
  • 08:09
    kayhayen closed #622
  • 07:55
    kayhayen review_request_removed #665
  • 07:55
    kayhayen review_request_removed #665
  • 07:55
    kayhayen review_requested #665
  • 07:55
    kayhayen review_requested #665
  • 07:55

    kayhayen on develop

    Add OpenBSD to clang and posix … (compare)

  • 07:55

    kayhayen on develop

    Add OpenBSD to clang and posix … (compare)

  • 07:55
    kayhayen closed #665
  • 07:55
    kayhayen closed #665
  • 07:35
    kayhayen commented #665
  • 07:35
    kayhayen commented #665
  • 07:34
    kayhayen milestoned #665
  • 07:34
    kayhayen milestoned #665
Kay Hayen
@kayhayen
(test suite running just fine)
I think I will merge this into my private main branch soon now, lets see how long it's from there to release. :)
herbal-rage
@herbal-rage
hey, can someone please hint me at how can i use PIL library with nuitka? it keeps on failing to import it, and i suspect its because the package is actually called pillow. I tried pyinstaller and cx_freeze but they do seem to work worse since a lot of other issues are risen. Thanks in advance.
Jacob Merrill
@BluePrintRandom
@herbal-rage - I think nuitka would work better if there were a way to just grab the uncompiled C from pil and absorb it.
(like pull the source of the lib you intend to bind into nuitka)
[could skip a bunch of steps I think as far as speed is concerned]
Kay Hayen
@kayhayen
Not happing, Nuitka is not allowing to mix with C code, ever. The design will not allow it.
Kay Hayen
@kayhayen
@herbal-rage the pil is supposed to work fine. if not, make a minimal reproducer and file an issue, but there are no known issues about it.
Kay Hayen
@kayhayen
Wow, tuple comparisons are tricky, just fixed another bug in it. But I believe, slowly the branch is getting there, this should be the majority of the fixes now.
Unfortunately it always takes forever to re-run tests...
Kay Hayen
@kayhayen
Good times here, the target type shape work is turning green. Pretty many regressions I had, esp. with the tuple compare, I wasn't good at implementing that. But anyway, it's there now. My only worry is the in-place operations. Nuitka does a lot more than it used to do, and they might well be incorrect now, or be regressed for performance, given that temporary variables take a reference. I think attribute, subscript, class variable in-place operations clearly have to suck still, but I am still convinced CPython is equally bad, but I have to hope that local variable and module variable didn't regress.
Kay Hayen
@kayhayen
So I was working on making the generated tests smooth, and they now come from jinja files, and use jinja for loops and stuff. I found that the operations test was doing all it said it would, so I added that, but it found nothing more. But when running debug mode, yet again it finds more complaints, e.g. about frozenset, or bytearray, so I added even more descriptions for the binary operations, these are very nearly complete for a large set of builtin types now, so that's great.
And now I added the correctness test for in-place... and well, I figure the @= in-place operation is not know to many a tool. My pylint says it's a syntax error, despite supporting @ as an operation.
Ok, now that I want to reproduce it, it works ?!
Kay Hayen
@kayhayen
So, in-place operations give huge amounts of warnings about missing operations now. Like e.g. BINARY_OPERATION_TRUEDIV_LIST_BYTES_INPLACE that are an indication of not having this recognized at run time.
I mean compile time.
Right now, the in-place operations don't provide type shapes, etc. and do not optimize at all. I am tempted to take a stab at it. At least to get to the point, where the exceptions are being detected. These are of course not optimizing real programs for anything but pylint level of checks, but they make sure I have a clean view on what's useful and what is not.
That should not be too much work, as += information for types can be derived from + very much. So this is all about adding the specialized node variants for in-place too, solving the issue that the finally all binary operations are specialized in the node tree already, should be only roughly the same work as was done with binary operations in the course already.
Kay Hayen
@kayhayen
Puh, got the in-place operations up to the level of normal operations. The means in my optimization example, I am not any longer dependent on writing i = i + 1 but i +=1 is actually working now too or should be. Going to verify now, that these operation [] += b'' now properly become exceptions and don't annoy code generation in my test.
I cleaned up base classes, and now it's clean. Also having per operation nodes for in-place stuff can only be good to save stuff. Using mixins to carry over special + (sequence concat) and * (sequence repeat) stuff reminds me, that these do not use slots yet. I wonder if one day, I am going to revisit the merging meta class. I mean, having base classes is not too harmful, but some prevent slots for no real reason. Being able to use less memory for operations is probably a big thing. But I do not want to detract right now. Get in touch with me, if you think you can do a meta class that merges methods into one dictionary while keeping overload semantics intact.
Kay Hayen
@kayhayen
Did you know that while [] + () won't work, with in-place operations, it does work? Also strings, bytes, etc. all sequences apparently
Kay Hayen
@kayhayen
So I changed in Nuitka all += [] to use += () now, just for micro-optization sake. However, this also put an interesting though into my mind, say we know that a is a list, then we probably should have a variant for code generation that just accepts a producer of values, but not a tuple in a += (x,b). Obviously optimization can change these to plain list appends, but in case of large ones, that's not flying for large lists, and it might loose hints, so I guess, it should rather be a.extend((x,b)) but for that extend helper, we don't want it to ever have the tuple build, but instead pass a producer, which could be a mere PyObject ** iterator with a size, that lives on the caller stack. But something like a generator, can also be used in-place with a list, and for that, I am getting back to quick iteration yet again. But with C types, we could avoid having that entirely.
Right now, Nuitka uses something like this:
static bool MAKE_QUICK_ITERATOR(PyObject *sequence, struct Nuitka_QuickIterator *qiter) {
    if (Nuitka_Generator_Check(sequence)) {
        qiter->iterator_mode = ITERATOR_COMPILED_GENERATOR;
        qiter->iterator_data.generator = (struct Nuitka_GeneratorObject *)sequence;
    } else if (PyTuple_CheckExact(sequence)) {
        qiter->iterator_mode = ITERATOR_TUPLE;
        qiter->iterator_data.tuple_data.tuple = (PyTupleObject *)sequence;
        qiter->iterator_data.tuple_data.tuple_index = 0;
    } else if (PyList_CheckExact(sequence)) {
        qiter->iterator_mode = ITERATOR_LIST;
        qiter->iterator_data.list_data.list = (PyListObject *)sequence;
        qiter->iterator_data.list_data.list_index = 0;
    } else {
        qiter->iterator_mode = ITERATOR_GENERIC;

        qiter->iterator_data.iter = MAKE_ITERATOR(sequence);
        if (unlikely(qiter->iterator_data.iter == NULL)) {
            return false;
        }
    }

    return true;
}
That is used only in BUILTIN_SUM1 so far. Mind you, we are using this merely to get away with the need to build any actual iterator as a Python object.
It makes sum with a generator work with acceptable speed. Such an iterator could of course also have a case for PyObject * arrays, as discussed above, and probably would be needed too, as we never can be entirely certain of what it is we are being called on. And e.g. with Nuitka generators, this avoids a few hoops to normally jump through, e.g. no preservation of the local execution context, as sum doesn't really need that.
Kay Hayen
@kayhayen
I guess, I will re-visit code generation once the operations are under control, and try to add the qiter as a target type, for the sum value, and see what that does. Having code generation of BUILTIN_SUM with a LIST type could remove the need, and imagine we know e.g. for sure, that it's a list of int, then we can expand further. Getting the QITER supported as a target type, in much the same way as LIST, TUPLE would be an interesting challenge and interesting demonstration, as well as pathing the way for in-place list add to become really great too. The challenge will be to know that list extend would really love a qiter, and to add this as a C target type, that normally only calls MAKE_QUICK_ITERATOR, but for others, e.g. tuple creation, does it more directly.
Right now, I still had a few regressions with the in-place add work. Classmethods and normal methods don't mix very well, but I hope to have it largely under control now.
Kay Hayen
@kayhayen
I am growing suspicious, that I have passed cls in cases, where it's taken as a self without being the object, and then it's not using the proper stuff. I might have to use singletons for type shapes of built-ins rather than their class. I guess, that's necessary to at least remove the doubt, even if works correctly now.
Ok, and now it's back to square one, as the reference count tests as finding in-valid objects with in-place operations. I thought that wasn't the case anymore. :(
Ok, stop dreaming. Unfortunately I will be busy with other things for the largest part today. So lets hope on this gets resolved soon.
Kay Hayen
@kayhayen
Yes, constants are propagated into in-place assignments, and then in-place assigning the result to a local variable or whatnot, is not really that. that ought to be possible to get under control though.
But suddenly a lot of things started crashing, where previously they were already working. I guess, it is because the now have type shapes and control flows that know the result, and these new optimization now need to be dealt with properly.
As usual, adding any optimization means there is a need to cleanup existing stuff and the technical debt it has, but I like it, that's great
Kay Hayen
@kayhayen
So, I figure it out, basically for in-place operations on constants, the issue was the const code doesn't give a reference, but the in-plaec helper expects it, and an in-place operation used to assign a value merely, was also not releasing the result then, but that is fixed now. I am running i over the build bots, and expect there to have this issue fixed, that only came up, because in-place operations were suddenly subject to optimization, which was never the case before.
There is a missing optimization, in that e.g. inplace add to a float, from what is known to be a type that is not having __radd__ overloaded, can be lowered to mere add.
Right now, we have a = operator.iadd(some_float, some_non_object). And we we know the shape well enough, we can go to mere + except that it might not be possible to add to floats, which would mention that in the error message. A small detail, potentially not worth bothering, but tests found that and old Nuitka actually did it for constant left values, to always demote to standard binary operation.
Kay Hayen
@kayhayen
I think now I want to finish up on the shape class and instance mix, and cleanup the mess, by using only instances. That may or may not sacrifice performance, but it's needed for clarify. Once that is done, and I think it's just a lot of editing work, one that search/replace can do a lot about though, I can put it on factory and develop shortly after.
That will be huge celebrations. From there, I guess, short term goals should round it up, e.g. making more suitable in-place templates that can create effective code with more right hand types.
And yes, I decided to cleanup the qiter mess of "sum" next then, not only because it will become even faster, but also because it paths the way towards loop variable optimization.
Kay Hayen
@kayhayen
so I did the mass change of shapes from classes/instance mix to instances only, and mostly dealt with the fallout of that.
While doing it, I noticed that complex constants were not using the complex shape, and of course, next thing I realize is that tests are failing, I neglected the ** ability to raise ZeroDivisionError and that made tests fail.
Then of course, error message for < changed for CPython36, which I noticed only now, and generally some minor optimizations for the raising of that exception for the binary operations get applied there too.
Right now I am making commits of the massive change, this is one of the biggest changes to Nuitka in a long time.
For comparisons, far too few types are rolled out at this time, comparison template isn't as good yet, as is in-place operations.
One this is out, there is a bunch of follow up actions too, where more of these things get tuned to the maximum.
Kay Hayen
@kayhayen
So, it's out on factory, with a bunch of extra improvements, e.g. now the boolean check of integer comparisons no longer thinks it could raise when checking for truth. This kind of thing can really reduce code.
Right now I am investigating my loop integer example for improvement opportunities. I also reminded myself that unary operations should be handle in the same fashion.
Kay Hayen
@kayhayen
Ok, I just looks at my looping integer example, and guess what, it longer detects the int or int or long type shape correctly, but puts an unknown there. I guess, that is because I had bugs before, that got fixed when giving up the instance, class mess.
But I had some thought during my morning sport routine, and came up with a much cleaner approach that ought to work. Now going over developer manual, and updating it where it's outdated, with a plan to describe the loop SSA plan that I formed there.