Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Konstantin Burlachenko
    @burlachenkok
    Hello, everybody. I have look into the slides https://www.dabeaz.com/python/UnderstandingGIL.pdf and video https://www.youtube.com/watch?v=Obt-vMVdM8s . I would like to ask a question - when I write programs for typical CPU (for me it's x86/x64, and aarch32/64 I care about "volatile" - type like qualifier of memory to not allow the compiler to optimize load of variable and memory fences (e.g. https://clang.llvm.org/doxygen/stdatomic_8h.html#a817535e09bf9c427a1cbee66d06e7030 intrinsic or another).
    Q1: Now if move to the level of Python programmer - do I need to specify a "volatile" qualifier for a list if it used by several threads (thread.Threading) in Python program if it is used as common preallocatted buffer for read/write from different threads (with locks). I did not see such qualifier in the language.
    Konstantin Burlachenko
    @burlachenkok
    Q2: How can I insert memory fences for threads? So David Beazley said that GIL lock is in some way will be release in I/O call. But at least for x86 the system calls happens with INT(https://www.felixcloutier.com/x86/intn:into:int3:int1) or SYSENTER (https://www.felixcloutier.com/x86/sysenter) and this command does not guarantee MFENCE instruction will be implicitly executed (https://www.felixcloutier.com/x86/mfence). So how in Python language insert Memory Fence?
    Kenneth Nielsen
    @KennethNielsen
    I'm not familiar with low level language implications of your question, but I can attempt to answer some of the question from the Python perspective. Per default, all data structures are accessible to all threads that have a reference to them and any kind of memory safety for shared data structures will have to be implemented on top. The one exception to this, and where GIL enters the picture is, that since there is only one interpreter with a global lock, it will only ever allow one thread to progress at a time, where the smallest chunk that a thread may progress is determined by the bytecode, which form atomic operations. What this means is that if the operations you wish to perform with a shared memory structure is atomic, i.e. consist of a single bytecode, than such interactions should be threadsafe even without local locking. Here is a reference to atomic operations in Python: http://larsyencken.github.io/gitbook-mcpy/data_structures/atomic.html. So as you can see from that list, you can e.g. use certain operations on lists and dicts, shared between threads, in a thread safe manner, without local locking.
    There are however at least two caveats to this answer, which I'm sorry to say that I don't know the details of. I remember there has been some talk of "thread local storage" (https://pythontic.com/multithreading/synchronization/thread-local-storage) which I'm unsure to what level is implemented and to what level it affects my reply about "all data structures are accessible to all threads" and there are also rumblings about "sub-interpreters", which I'm fairly certain is not yet a released feature, but which will probably affect the "that since there is only one interpreter with a global lock" parts of my reply. The upside to that is that I don't you will accidentally use, so my reply I think will continue to apply to the default mode of operation.
    Kenneth Nielsen
    @KennethNielsen
    All in all, I'm fairly certain the answer to Q1 is "no, you do not need to, but you need to be careful how you use it from the different threads so that it stays memory safe". The other option is to replace it with a data structure that is inherently thread safe like a Queue.
    Konstantin Burlachenko
    @burlachenkok
    Thanks @KennethNielsen !
    Konstantin Burlachenko
    @burlachenkok
    @KennethNielsen I would like to ask, why do you think about that "Per default, all data structures are accessible to all threads that have a reference to them and any kind of memory safety for shared data structures will have to be implemented on top"
    To implement that garantee the C++ implementation should be reentarable of standard types? Is it so for different instances of "lists"?
    Also I do not immeditely see usage of volatile keywoard during working with pointers ... https://github.com/python/cpython/blob/main/Objects/listobject.c
    If CPython will be compiled in such way that some fetching from memory will be putted into register....
    And absence of volatile will allow that...
    Konstantin Burlachenko
    @burlachenkok
    Then from the level of Python virtual machine, I will not have the ability to express "volatile" - like semantics...
    I'm now a bit scared about Python interpreter usage for multithreading at all, whatever the popularity of Python is.
    David Beazley
    @dabeaz
    Would suggest searching the Python bug/issue tracker for the word "volatile" to see if this has ever been reported as any kind of issue.
    Joshua Bronson
    @jab
    https://github.com/faster-cpython/ideas/discussions/328 neat (+ lol @ "a guy named Dave Beazley")
    David Beazley
    @dabeaz
    Oh. So that explains the sudden emails I'm getting on a long-gone issue. If there ever was a prototypical example for "perfect being the enemy of the good", it would definitely be that whole thing though. LOL.
    Don Hopkins
    @SimHacker
    Hi David! I just mentioned your secret vault talk on HN, which I enjoyed years ago and frequently recommend other people check out. Now I'm treating myself to watching another one of your talks, the Fun of Reinvention, the one with the guinea pig riding a skateboard. I'm still having fun pushing piles of Python today of course, both writing new code and maintaining old code (which is still enjoyable because Python's so easy to read, and I've been polishing it decades). And I agree that f-strings are great, even though I was skeptical at first. But now that I've seen f-strings in f-strings my mind is re-blown! I just wanted to tell you how much I admire your work and thank you for all those great talks. I hope I can see one in person some day once it's safe to go to conferences again, and hopefully you will visit Amsterdam to talk about Python. And yes, I sure remember "xv"! That was quite an awesome tool in its time. https://news.ycombinator.com/item?id=30765748
    Don Hopkins
    @SimHacker
    Claiming "super() is the glue that holds components together" is turning object oriented programming on its side! By 90 degrees, so up is to the right.
    How about a meta-meta-object-protocol that gives each class its own 2d transformation, so you can rotate 180 degrees to inherit from your subclasses? ;)
    Shukurali
    @ShukurDev
    Hi
    Joshua Bronson
    @jab
    AlirezaNorouzi
    @alirezan81

    If there is an unauthorized token in the code, at what stage does the syntax error related to the unauthorized token occur and the program stops? lexing or parsing?

    In some articles it is written that because the job of the lexer is to separate the permissible and meaningful tokens of the language. At this point it gives an error and the program stops.

    Elsewhere it is written that a unauthorized token is considered unknown and is transferred to the next stage of decomposition and gives an error.

    Konstantin Burlachenko
    @burlachenkok
    I don't have proper compiler background, but from my CS education point of view - Lexical analysis of the program is splitting the program into tokens. And Parser identifies language elements (The token can be of the following type (Operators, Identifiers, Keywords, Constants). Rules of the writing that elements can be pretty different (0x123 is a valid integer constant in C/C++ and 0QWE is not valid identifier for variable, class, etc.). So I bet such checking should be done by Parser. @alirezan81
    2 replies
    AlirezaNorouzi
    @alirezan81

    Is not the job of the lexer to separate meaningful tokens?
    So how do you recognize and create a token that makes no sense?
    5 $ 4
    5 = number
    $ =?
    4 = number

    In addition, in the principles of compiler design, we have three types of errors: lexer error, parser error and semantic error. But in Python I do not know if this is the case or not because it was different in some parts @burlachenkok

    David Beazley
    @dabeaz
    A symbol like $ is either a token or it's not. If it's not, then it's an error that would be reported in the lexer.
    AlirezaNorouzi
    @alirezan81
    So at this stage the lexer gives an error and the program stops, and does not go to the next stage, parsing. It is true? @dabeaz
    David Beazley
    @dabeaz
    Are you asking me about Python specifically or compiler writing in general? The answer is it depends.
    AlirezaNorouzi
    @alirezan81
    The compiler, which is clear that each step has its own error, and the token error is in the lexing.
    I'm asking about the cpython interpreter @dabeaz
    David Beazley
    @dabeaz
    I wouldn't know on cpython. Plus, a lot has changed since the last time I looked at it (with the PEG parser and other additions).
    Jiqing Tang
    @jiqingtang
    hi @dabeaz I signed up for one of your courses and have some questions (sent to your email), can you take a look and get back to me when you have time? thanks!
    Mustafa Cenk Aydın
    @CenkAydin
    Hello everyone! I'm making a project. It's a project like a drawing robot. But not robot, it's just a GUI app. For example: L 36 [L 4 [F 100 R 90] R 10] in this example F means draw n step forward, R means turn n degrees to right, and L means repeat the parantheses n times(loop). I separated this into tokens with lex. Now i have to parse with yacc. But i'm keep taking errors with that. I read Yacc example for calculator in this site. But i couldn't adapt it to my own code. Can you help me with that?
    Joshua Bronson
    @jab
    And now for something completely different:
    https://twitter.com/AmazingThew/status/1533864962216669185
    1 reply
    pure genius amirite
    Joshua Bronson
    @jab
    Joshua Bronson
    @jab
    (Thanks to Dave for putting it on my radar:)
    Thomas Grainger
    @graingert_gitlab
    Oh I didn't know there was a dabeaz room
    I was messing around with curio recently and had a go at running a sidecar coroutine while an outer coroutine was suspended
    I'm not going to do anything with it, but it has the upshot of being able to set an event with a synchronous function and then queue them all up and run them the next time the outer coro is suspended
    graingert
    @graingert:matrix.org
    [m]
    now the big idea is if curio grew an io submission queue
    graingert
    @graingert:matrix.org
    [m]
    so you could do
    queue = await curio.submission_queue()
    
    queue.submit(CurrentTask())
    queue.submit(WriteFile(file, data))
    wrote = None
    while (result := queue.pop_nowait()):
        match result:
            case CurrentTask(current_task): ...
            case WriteFile(wrote)
    if wrote is None:
        result = await queue.pop()
    so here the curio kernel has an opportunity to run code in the submit and pop_nowait functions
    so it's pretty easy to see how a synchronous operation like CurrentTask might work
    however some io multiplexer interfaces don't support pushing completions to a queue without you requesting them, eg select/epoll
    so you'd have to run a select/poll with 0 timeout when you run out of completions in pop_nowait(), so you return None if there's no completions in that 0 timeout, then the async function has to call await queue.pop() to wait for the next completion
    but the cool thing is when you switch to an OS kernel interface that does support pushing completions to your queue
    graingert
    @graingert:matrix.org
    [m]
    but there's loads of places where completions can go on the queue without awaiting - eg a thread can push to a deque and wake a task soon, but if the task is already awake then it doesn't need to
    graingert
    @graingert:matrix.org
    [m]
    actually if all the other tasks were idle you'd be able to run the selector for the full timeout to the next timer
    Thomas Grainger
    @graingert_gitlab
    Here's an implementation of an async function called "become_twisted()" when awaited it makes the curio function into a twisted async function. I highly suspect this was actually the intended design considering the comment on "curio.Task.send"
    https://gist.github.com/graingert/120e3f9c9d9616897ca7c00d55ecfb85#file-become_twisted-py-L39