Hello, everybody. I have look into the slides https://www.dabeaz.com/python/UnderstandingGIL.pdf and video https://www.youtube.com/watch?v=Obt-vMVdM8s . I would like to ask a question - when I write programs for typical CPU (for me it's x86/x64, and aarch32/64 I care about "volatile" - type like qualifier of memory to not allow the compiler to optimize load of variable and memory fences (e.g. https://clang.llvm.org/doxygen/stdatomic_8h.html#a817535e09bf9c427a1cbee66d06e7030 intrinsic or another).
Q1: Now if move to the level of Python programmer - do I need to specify a "volatile" qualifier for a list if it used by several threads (thread.Threading) in Python program if it is used as common preallocatted buffer for read/write from different threads (with locks). I did not see such qualifier in the language.
Q2: How can I insert memory fences for threads? So David Beazley said that GIL lock is in some way will be release in I/O call. But at least for x86 the system calls happens with INT(https://www.felixcloutier.com/x86/intn:into:int3:int1) or SYSENTER (https://www.felixcloutier.com/x86/sysenter) and this command does not guarantee MFENCE instruction will be implicitly executed (https://www.felixcloutier.com/x86/mfence). So how in Python language insert Memory Fence?
I'm not familiar with low level language implications of your question, but I can attempt to answer some of the question from the Python perspective. Per default, all data structures are accessible to all threads that have a reference to them and any kind of memory safety for shared data structures will have to be implemented on top. The one exception to this, and where GIL enters the picture is, that since there is only one interpreter with a global lock, it will only ever allow one thread to progress at a time, where the smallest chunk that a thread may progress is determined by the bytecode, which form atomic operations. What this means is that if the operations you wish to perform with a shared memory structure is atomic, i.e. consist of a single bytecode, than such interactions should be threadsafe even without local locking. Here is a reference to atomic operations in Python: http://larsyencken.github.io/gitbook-mcpy/data_structures/atomic.html. So as you can see from that list, you can e.g. use certain operations on lists and dicts, shared between threads, in a thread safe manner, without local locking.
There are however at least two caveats to this answer, which I'm sorry to say that I don't know the details of. I remember there has been some talk of "thread local storage" (https://pythontic.com/multithreading/synchronization/thread-local-storage) which I'm unsure to what level is implemented and to what level it affects my reply about "all data structures are accessible to all threads" and there are also rumblings about "sub-interpreters", which I'm fairly certain is not yet a released feature, but which will probably affect the "that since there is only one interpreter with a global lock" parts of my reply. The upside to that is that I don't you will accidentally use, so my reply I think will continue to apply to the default mode of operation.
All in all, I'm fairly certain the answer to Q1 is "no, you do not need to, but you need to be careful how you use it from the different threads so that it stays memory safe". The other option is to replace it with a data structure that is inherently thread safe like a Queue.
To implement that garantee the C++ implementation should be reentarable of standard types? Is it so for different instances of "lists"?
Also I do not immeditely see usage of volatile keywoard during working with pointers ... https://github.com/python/cpython/blob/main/Objects/listobject.c
If CPython will be compiled in such way that some fetching from memory will be putted into register....
And absence of volatile will allow that...
I'm now a bit scared about Python interpreter usage for multithreading at all, whatever the popularity of Python is.
Hi David! I just mentioned your secret vault talk on HN, which I enjoyed years ago and frequently recommend other people check out. Now I'm treating myself to watching another one of your talks, the Fun of Reinvention, the one with the guinea pig riding a skateboard. I'm still having fun pushing piles of Python today of course, both writing new code and maintaining old code (which is still enjoyable because Python's so easy to read, and I've been polishing it decades). And I agree that f-strings are great, even though I was skeptical at first. But now that I've seen f-strings in f-strings my mind is re-blown! I just wanted to tell you how much I admire your work and thank you for all those great talks. I hope I can see one in person some day once it's safe to go to conferences again, and hopefully you will visit Amsterdam to talk about Python. And yes, I sure remember "xv"! That was quite an awesome tool in its time. https://news.ycombinator.com/item?id=30765748
How about a meta-meta-object-protocol that gives each class its own 2d transformation, so you can rotate 180 degrees to inherit from your subclasses? ;)
If there is an unauthorized token in the code, at what stage does the syntax error related to the unauthorized token occur and the program stops? lexing or parsing?
In some articles it is written that because the job of the lexer is to separate the permissible and meaningful tokens of the language. At this point it gives an error and the program stops.
Elsewhere it is written that a unauthorized token is considered unknown and is transferred to the next stage of decomposition and gives an error.
I don't have proper compiler background, but from my CS education point of view - Lexical analysis of the program is splitting the program into tokens. And Parser identifies language elements (The token can be of the following type (Operators, Identifiers, Keywords, Constants). Rules of the writing that elements can be pretty different (0x123 is a valid integer constant in C/C++ and 0QWE is not valid identifier for variable, class, etc.). So I bet such checking should be done by Parser. @alirezan81
2 replies
Is not the job of the lexer to separate meaningful tokens?
So how do you recognize and create a token that makes no sense?
5 $ 4
5 = number
$ =?
4 = number
In addition, in the principles of compiler design, we have three types of errors: lexer error, parser error and semantic error. But in Python I do not know if this is the case or not because it was different in some parts @burlachenkok
Hello everyone! I'm making a project. It's a project like a drawing robot. But not robot, it's just a GUI app. For example: L 36 [L 4 [F 100 R 90] R 10] in this example F means draw n step forward, R means turn n degrees to right, and L means repeat the parantheses n times(loop). I separated this into tokens with lex. Now i have to parse with yacc. But i'm keep taking errors with that. I read Yacc example for calculator in this site. But i couldn't adapt it to my own code. Can you help me with that?
And now for something completely different:
https://twitter.com/AmazingThew/status/1533864962216669185
https://twitter.com/AmazingThew/status/1533864962216669185
1 reply
pure genius amirite
I was messing around with curio recently and had a go at running a sidecar coroutine while an outer coroutine was suspended
I'm not going to do anything with it, but it has the upshot of being able to set an event with a synchronous function and then queue them all up and run them the next time the outer coro is suspended
https://gist.github.com/graingert/00798d4d60ca080f1b1080e5d311b757#file-gimcrack-py-L146 it's a mess and I made it a year ago
so you could do
queue = await curio.submission_queue()
queue.submit(CurrentTask())
queue.submit(WriteFile(file, data))
wrote = None
while (result := queue.pop_nowait()):
match result:
case CurrentTask(current_task): ...
case WriteFile(wrote)
if wrote is None:
result = await queue.pop()
so here the curio kernel has an opportunity to run code in the submit and pop_nowait functions
so it's pretty easy to see how a synchronous operation like CurrentTask might work
however some io multiplexer interfaces don't support pushing completions to a queue without you requesting them, eg select/epoll
so you'd have to run a select/poll with 0 timeout when you run out of completions in pop_nowait(), so you return None if there's no completions in that 0 timeout, then the async function has to call await queue.pop() to wait for the next completion
but the cool thing is when you switch to an OS kernel interface that does support pushing completions to your queue
but there's loads of places where completions can go on the queue without awaiting - eg a thread can push to a deque and wake a task soon, but if the task is already awake then it doesn't need to
actually if all the other tasks were idle you'd be able to run the selector for the full timeout to the next timer
Here's an implementation of an async function called "become_twisted()" when awaited it makes the curio function into a twisted async function. I highly suspect this was actually the intended design considering the comment on "curio.Task.send"
https://gist.github.com/graingert/120e3f9c9d9616897ca7c00d55ecfb85#file-become_twisted-py-L39
https://gist.github.com/graingert/120e3f9c9d9616897ca7c00d55ecfb85#file-become_twisted-py-L39