Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen labeled #493
  • 11:54
    kayhayen unlabeled #493
  • 11:54
    kayhayen unlabeled #493
  • 11:54
    kayhayen edited #493
  • 11:54
    kayhayen edited #493
  • 11:53
    kayhayen commented #493
  • 11:53
    kayhayen commented #493
  • 11:47
    kayhayen labeled #479
  • 11:47
    kayhayen labeled #479
  • 09:24
    golyalpha commented #493
  • 09:24
    golyalpha commented #493
  • 09:21
    golyalpha commented #493
  • 09:21
    golyalpha commented #493
  • Aug 19 21:40
    tommyli3318 commented #495
  • Aug 19 21:40
    tommyli3318 commented #495
Jorj X. McKie
@JorjMcKie
@kayhayen - power failures continue to exist, but it recently has improved. Did I mention that people here are telling me that I am living in a privileged area anyway? Because of a nearby military hospital, that part of the island is only rarely hit with massive power outages ... that was good to know, a real help to regain emotional balance. Now it seems I am paying back those advantages with failures of several electrical devices (central water pump, refrigerator, air conditioner). Presumable those failures occurred because of the oscillating power voltages (vulgo: when power comes back, it may be much more than the tolerable 110V or 220V). Need I say more?
Kay Hayen
@kayhayen
@JorjMcKie I was thinking so, apparently managing a power grid is not trivial in the best of all times, but I am glad you are back on for now :)
Kay Hayen
@kayhayen
I was working on TODOs related to nodes, I made sure that all nodes that have a single child, use the specialization that is designed for that, since it allows __slots__ making way for less memory usage and slightly better Nuitka compilation times. This also cleaned up inheritances, and made implementation provision into a mixin rather than a base class, making for a cleaner design side of things too.
Kay Hayen
@kayhayen
There is a fixup on factory branch for standalone with recent Anaconda for Python3.
Seems their bdist_conda is severely broken.
Abhinav Sukumar Rao
@AetherPrior
Hello guys!
I'm Abhinav, an undergrad student from India. I am interested in what you're working on
After emailing @kayhayen , I was told about C helper codes.
However I wasn't exactly able to understand what that meant
Kay Hayen
@kayhayen
Welcome on board @AetherPrior good to have you here.
I was just going to write about what I got done today, so one of the re-formulations of Nuitka is that function bodes of all kinds, have this try/finally where they release parameter and user variables. They have been provided by the caller, parameter variables that is, and therefore we need to release them.
However, most of the time, code doesn't assign to the parameter variables at all, and so this lead to many nodes in our abstract execution tree, that were not really going to change or contribute much, and now I have added auto_releases for function nodes, where an analysis will put such parameter variables, often allowing to remove the try nodes, and generating simpler C code in the end. That is good for compile time memory usage of Nuitka, and the C compiler, and I wouldn't be surprised if it generated faster binary too. I am working on scalability right here.
Kay Hayen
@kayhayen
It will, for empty functions allow a full reduction of the function body to return None even with arguments. And then my intention is, to do something I started before, to generate not different code for all those empty functions with just pass but shared ones. There are modules in Python land, some of which have hundreds of functions defined that do nothing, but through decorators serve as devices to call other code. For now, these will suddenly become not the GB using mosters in C, but smaller. Of course, cache locality and other things will benefit if there is only one function being called that does nothing but release parameters, but it's always the same.
Abhinav Sukumar Rao
@AetherPrior
So from what I gather here, you're trying to make all those pass functions will now be a single, shared function instead of each being a function on its own.
*to be
Kay Hayen
@kayhayen
Yes, there is an incredible amount of those in some places. And generating C code for all of them, is a huge overhead, and not beneficial to performany.
I think you said, you have C++ experience, but you are aware that Nuitka roduces goto ridden C code, right?
It's most about C code generation, from a Python structure, Python is really the large part of it.
Abhinav Sukumar Rao
@AetherPrior
No, actually. I was unaware that it actually produced goto ridden code, although I did figure it was the case.
Kay Hayen
@kayhayen
As far as Python goes, we do one code base that works with 2.7 and 3.7 alike, so we don't use the cool new language features, although we implement them :)
Abhinav Sukumar Rao
@AetherPrior
I do however understand that it's mostly about c code generation
Kay Hayen
@kayhayen
Code is for a def f(a): return a now like this:
static PyObject *impl___main__$$$function_1_f( struct Nuitka_FunctionObject const *self, PyObject **python_pars )
{
    // Preserve error status for checks
#ifndef __NUITKA_NO_ASSERT__
    NUITKA_MAY_BE_UNUSED bool had_error = ERROR_OCCURRED();
#endif

    // Local variable declarations.
    PyObject *par_a = python_pars[ 0 ];
    PyObject *tmp_return_value = NULL;

    // Actual function body.
    CHECK_OBJECT( par_a );
    tmp_return_value = par_a;
    Py_INCREF(tmp_return_value);
    goto function_return_exit;

    // Return statement must have exited already.
    NUITKA_CANNOT_GET_HERE( __main__$$$function_1_f );
    return NULL;


function_return_exit:
   // Function cleanup code if any.
    CHECK_OBJECT(par_a);
    Py_DECREF(par_a);

   // Actual function exit with return value, making sure we did not make
   // the error status worse despite non-NULL return.
   CHECK_OBJECT( tmp_return_value );
   assert( had_error || !ERROR_OCCURRED() );
   return tmp_return_value;
}
Code gets a compiled function object self not to be confused with any such variables, and an array pointer with the arguments, which are then assigned.
Checkout CodeTemplatesFunctions.py to get a feel for how the blocks are assembled. It's using mere standard % style. I never convinced myself to like how format works. And I am contemplating to switch to Jinja2 templates, which we have been using to generate C code for specialized functions, e.g. adding a LONG and a FLOAT object.
Abhinav Sukumar Rao
@AetherPrior
Okay, I'll have a look at that file then
Kay Hayen
@kayhayen
Types other than PyObject * are very rare still, I am working on that, but right now I like to coach a lot too. You are welcome in this too.
One other thing is, that the node tree has a representation in XML if you use --xml with Nuitka, and it's giving you the struture:
    <node kind="ExpressionFunctionBody" line="1" co_kwonlyargcount="0" ps_name="f" ps_normal_args="a" co_argcount="1" co_posonlyargcount="0" co_new_locals="True" co_is_optimized="True" co_has_starlist="False" co_has_closure="False" ps_kw_only_args="" ps_default_count="0" co_lineno="1" co_has_stardict="False" co_varnames="a" name="f" co_name="f" ps_dict_star_arg="" co_filename="/home/hayen/repos/Py2C/Mini.py" code_flags="" ps_list_star_arg="" provider="__main__" co_kind="Function" flags="None">
      <role name="body">
        <node kind="StatementsSequence" line="1">
          <role name="statements" type="list">
            <node kind="StatementReturn" line="2">
              <role name="expression">
                <node kind="ExpressionVariableRef" line="2" owner="__main__$$$function_1_f" variable_name="a"/>
              </role>
            </node>
          </role>
        </node>
      </role>
Structural optimization is also cool. Finding a case, where some node could decide to replace itself with something else that is simpler and still work the same, that is optimization. There is plenty of room for that.
That node tree is normally in memory, and this is a dump of it. We might use it for caching results later. But for now I use to e.g. see that the Try is gone, and enjoy this as big progress. :)
Which part of India are you from @AetherPrior ?
Abhinav Sukumar Rao
@AetherPrior
Woah I'll really have to check all of this out ... I'll look at some today and some in the morning (it's midnight here) :)
Kay Hayen
@kayhayen
Just to give you some stuff to have an idea, what's there in ieces, and yeah. look at stuff, and find a spot that you want to play around with, ask lots of questions, I will hopefully be able to help
Abhinav Sukumar Rao
@AetherPrior
I'm from South India, more specifically Tamil Nadu, but I'm studying in Hyderabad
Kay Hayen
@kayhayen
If you would like small contained tasks, that is also OK, but I prefer to let you wonder and make your decisions.
big place :-)
I am in Karlsruhe Germany right now, although my heart will always be in Hamburg. :)
We don't have these big cities, Karlsruhe is medium city for Germany, 200k roughly
Abhinav Sukumar Rao
@AetherPrior

I am actually pretty new to open source contribution, so I'm not very confident about large tasks as of yet :)

I am in Karlsruhe Germany right now, although my heart will always be in Hamburg. :)

Good to know :)
Hyderabad is actually a state capital, so that's why it's pretty big.
Population is large ~6.8m (well it's India)

Kay Hayen
@kayhayen
yup, read that. Don't worry, there is not a lot that can go wrong, except if you despair from the difficulty that sometimes comes up, when something needs yet another twist to be correct, it can be at times frustrating, but on the other hand more rewarding when then it works
Abhinav Sukumar Rao
@AetherPrior
I have a question,
The files present in templates are the ones that contain C code templates, and are the ones you'd like to convert to Jinja2 format right?
Also the files present in "codegen" outside of templates, are the ones using these templates and filling the placeholder strings, right?
Kay Hayen
@kayhayen
yes, sometimes small template strings are in codegen as well, not everything was really made into a template. I think e.g. that Jinja2 is very much better at block and identation in general. I am using some for specialization, search for *.j2 in the repo, and I liked it a lot more, however I never made up my mind, if we even need to this to be fast, and if it is slower or faster for the things we do.
But real program code generation is all directly below nuitka.codegen @AetherPrior .
Abhinav Sukumar Rao
@AetherPrior
Understood
Kay Hayen
@kayhayen
Thanks for taking care of #365 @AetherPrior saves me the time, very much appreciated. :)
In other news, I just pushed to github develop from my factory and jorj, and rebased jorj to the result @JorjMcKie
On factory my auto-release for parameter variables and shared empty functions code, lives until I convinced myself that function inlining makes sure to add back releases. So I think it might right now have that problem, so it's not yet ready, but I think I might get there.
The donations paid machine failed when I attempted to upgrade it, turns out the SD card was worn out. I got a new one and will install it. I will need it for making releases. Otherwise I surely will make a pre-release. Luckily installing it is automated with Ansible, so now that I have the replacement SD card next to me, that's probably surely happening this weekend.
Kay Hayen
@kayhayen
I got the donations paid machine (donatix) up and running again, but it's still compiling all the Pythons, so most of the stuff is on factory only.
Interesting stuff is that factory now uses Clang as installed by MSVC 2019 if it is there. Unfortunately I am only seeing a 32 bits target included. I was hoping for easy way to install it.
Kay Hayen
@kayhayen
Overall the Debian Buster migration is kind of shaking up my environment a bit.