These are chat archives for dropbox/pyston

12th
Aug 2015
Sun
@Daetalus
Aug 12 2015 02:23

Hi, for sys.float_info and similar things. I did some investigation.

One way is enable PyStructSequence_New and related things. But need to apply some "Pyston Changes". This way still need some investigation.

The other way is use new BoxedClass, such as the current BoxedSysFlags did, but need some(maybe lot) improvements. Maybe need to write a new base class, for example, aNamedTuple class for potential **_info class. To consistent with CPython behaviors.

Which way is more acceptable?

Kevin Modzelewski
@kmod
Aug 12 2015 02:27
oh hmm I thought we had PyStructSequence already
I don't know much about this particular case, but I think in general we should prefer to do things exactly the way CPython does, unless there's a specific reason not to
Sun
@Daetalus
Aug 12 2015 02:29
Yes, but not really work yet. At least for sys.
Ok
Kevin Modzelewski
@kmod
Aug 12 2015 02:29
it sounds like the copy-CPython approach might be more work up-front, but I think it's better in the long run
(unless there's some reason we need this quickly)
since other things will probably want it in the future
Sun
@Daetalus
Aug 12 2015 02:30
Got it.
Thanks!
Sun
@Daetalus
Aug 12 2015 03:28

@kmod , PyStructSequence is a public API. And if some one want to create a namedtuple in C level. They will create PyTypeObject in this way:

static PyTypeObject FooResultType = {0, 0, 0, 0, 0, 0};

But in pyston::BoxedClass, this is protected constructor.

Any suggestion?

Kevin Modzelewski
@kmod
Aug 12 2015 03:36
does it work to do
static struct _typeobject instead of static PyTypeObject?
Sun
@Daetalus
Aug 12 2015 03:37
yes, it could work. But it is a public CAPI...
Kevin Modzelewski
@kmod
Aug 12 2015 03:38
well, there's some funky stuff if you try to use it inside a pyston file
normally PyTypeObject refers to _typeobject, but if we detect that we're compiling Pyston, we change it to refer to BoxedClass
Sun
@Daetalus
Aug 12 2015 03:38
Oh, I see...
Seems the parameters match has problem. Working on it.
Kevin Modzelewski
@kmod
Aug 12 2015 03:41
yeah, this might be tricky to do inside C++
another option is to compile it as a C file which gets the normal CAPI rules
Vinzenz Feenstra
@vinzenz
Aug 12 2015 09:16
@tjhance well I figured out what I did wrong. Now the whole things is much more readable. So the slice type implementation PR is coming up today
@undingen Congrats on the position at Dropbox :-)
I haven't been much following that, but I already suspected that you're working there when I saw the amount of commits from your side :D
Sun
@Daetalus
Aug 12 2015 10:18

Two questions:

  1. In CPython, complex will check whether the arguments is number, through x->cls->tp_as_number. But in Pyston, if my understanding was correct, the cls->tp_as_number always not NULL.

  2. use sys_module->giveAttr("xxx_info", function_return_Py_SequenceStructure()) will get "not valid GC Object error". Does that means Pyston could not handle struct?

Kevin Modzelewski
@kmod
Aug 12 2015 10:26
if we have C code that statically declares an object
we need to call the PyGC_AddNonheapRoot function on it
before passing it off to pyston code
(or if it's a PyTypeObject, call PyType_Ready which will call AddNonheapRoot for uss)
for #1, does it check just tp_as_number, or does it look at one of the functions specifically?
I think you're right that we have tp_as_number more often than cpython does, but I think the individual fields of it will be filled out the same way
Sun
@Daetalus
Aug 12 2015 10:30

This is in CPython:

    nbr = r->ob_type->tp_as_number;
    if (i != NULL)
        nbi = i->ob_type->tp_as_number;
    if (nbr == NULL || nbr->nb_float == NULL ||
    ...

This is my implementation:

(_real->cls->tp_as_number == NULL && 
!_real->cls->getattr(float_str))
And I also tried to copy CPython implementation, same result.
r and _real is the real part of complexNew's arguments.
Kevin Modzelewski
@kmod
Aug 12 2015 11:31
oh hmm so nb_float is getting set when it shouldn't be?
Vinzenz Feenstra
@vinzenz
Aug 12 2015 11:37
@kmod after implementing in pypa the handling for literal assignment errors and changing the expectation in the assign_literal test I get this:
6:                    assign_literal.py    FAILED (Exited with code -6 (expected code 1))
6: Traceback (most recent call last):
6:   File "/home/vfeenstr/pyston/build/Debug/src/codegen/parse_ast.py", line 183, in <module>
6:     m = compile(s, fn, "exec", _ast.PyCF_ONLY_AST)
6:   File "/home/vfeenstr/pyston/test/tests/assign_literal.py", line 2
6:     2 = 2
6: SyntaxError: can't assign to literal
6: pyston: ../../src/codegen/parser.cpp:1135: std::vector<char> pyston::_reparse(const char *, const std::string &, pyston::AST_Module *&): Assertion `code == 0' failed.
Oh I see, this happens with the CPython backend
Kevin Modzelewski
@kmod
Aug 12 2015 11:39
oh, yeah it doesn't handle syntaxerrors very well
Vinzenz Feenstra
@vinzenz
Aug 12 2015 11:40
Also I am trying to fix some other issues with the handling of complex numbers and negation
Seems like that those are actually not represented as unary expression
in CPython
Kevin Modzelewski
@kmod
Aug 12 2015 11:41
ok, nice :)
Sun
@Daetalus
Aug 12 2015 11:41
@kmod no, just tp_as_number always not NULL,
Vinzenz Feenstra
@vinzenz
Aug 12 2015 11:43
@Daetalus could you check if that is also the case when libpypa is not used? Might be a problem of my parser (just a guess)
ok nevermind, I just see that's not the AST
Sun
@Daetalus
Aug 12 2015 11:45
Thanks, but I believe it is not related. This is just because all cls is the instance of BoxedClass.
Vinzenz Feenstra
@vinzenz
Aug 12 2015 11:47
@kmod also seems like the thread_spawning.py tests are failing randomly
5:                   thread_spawning.py    FAILED (Exited with code -6 (expected code 0))
5: pyston: ../../src/gc/heap.cpp:625: pyston::gc::GCAllocation *pyston::gc::SmallArena::_alloc(size_t, int): Assertion `!myblock->prev' failed.
I also saw somewhere before that this is failing with some timeout
Sun
@Daetalus
Aug 12 2015 11:50
I ignored the timeout issue. (BTW, I just a volunteer, so don't take it as an official suggestion).
Vinzenz Feenstra
@vinzenz
Aug 12 2015 11:52
I am just letting them know that i am running into this.
I am trying to get my libpypa changes integrated, and trying to fix surrounding errors
Some of those errors just don't make sense
Vinzenz Feenstra
@vinzenz
Aug 12 2015 12:00
@kmod I'd be glad if you could explain me why this potentially would fail like this:
5:            exceptions_nonreprable.py    FAILED (bad output)
5: --- /tmp/expected_Vc2UvW    2015-08-12 13:59:22.688149613 +0200
5: +++ /tmp/received_7nQe5b    2015-08-12 13:59:22.688149613 +0200
5: @@ -1,7 +1,3 @@
5:  str
5:  
5:  str
5: -str
5: -str
5: -str
5: -str
5: 
 ./pyston_dbg test/tests/exceptions_nonreprable.py
str

Traceback (most recent call last):
  File "test/tests/exceptions_nonreprable.py", line 14, in <module>:
    raise BadException()
__main__.BadExceptionstr
and using -x it's the same
This is really something weird
The only difference I could see was when using cpython, there's no 'colon' after <module> in the traceback
That's the last test failing for me
not sure how to workaround this now
Vinzenz Feenstra
@vinzenz
Aug 12 2015 12:07
Anyway I have uploaded the latest changes
@kmod and thanks for the fedora build fixes
Sun
@Daetalus
Aug 12 2015 12:08
@vinzenz Do you have plan to add Python 3 support?
Vinzenz Feenstra
@vinzenz
Aug 12 2015 12:30
@Daetalus at some point for sure, right now I'd like the necessities of pyston to be covered, since that was the original reason for the parser in the first place.
Vinzenz Feenstra
@vinzenz
Aug 12 2015 12:56
awesome it looks finally good now :-)
Vinzenz Feenstra
@vinzenz
Aug 12 2015 13:14
@kmod I'd be glad if you can have a look at #826 if that looks ok to you
Vinzenz Feenstra
@vinzenz
Aug 12 2015 13:38
@kmod @undingen Just FYI I have just added support for detecting continue and break statements outside of loops to pypa
Vinzenz Feenstra
@vinzenz
Aug 12 2015 14:57
I am also now disallowing continue inside of finally blocks
Vinzenz Feenstra
@vinzenz
Aug 12 2015 15:50
Those threading tests are starting to annoy me with their random failures
Sun
@Daetalus
Aug 12 2015 17:33
Oh my god... one city in China just had a huge explosion. Like A-Bomb...
Rudi Chen
@rudi-c
Aug 12 2015 17:35
Coal mine?
Sun
@Daetalus
Aug 12 2015 17:35
Chemical warehouse, in Tianjing habor
Rudi Chen
@rudi-c
Aug 12 2015 17:36
Ah found it. It wasn't even showing up on Google News if I searched only for "china explosion".
Yeah that's a lot of smoke and fire.
Sun
@Daetalus
Aug 12 2015 17:38
You should watch the video... a huge, huge , huge explosion... Very closed to residential area.
Rudi Chen
@rudi-c
Aug 12 2015 17:39
It's more than twice the height of the nearby skyscrapers wow.
Marius Wachtler
@undingen
Aug 12 2015 18:06
:-(
Kevin Modzelewski
@kmod
Aug 12 2015 19:48
oh wow, that was huge
Travis Hance
@tjhance
Aug 12 2015 19:48
link?
Kevin Modzelewski
@kmod
Aug 12 2015 19:49
@vinzenz sorry for the delay, are you still running into any of those issues?
we've been having issues with the threading tests for a while
they never seem to manifest for me :(
Vinzenz Feenstra
@vinzenz
Aug 12 2015 21:11
@kmod Well look at the unit tests on travis ci for my PRs there it fails and its not because of what i did.
Kevin Modzelewski
@kmod
Aug 12 2015 21:19
yeah we're having some issues with that :(