https://github.com/aio-libs The set of asyncio-based libraries built with high quality
@asvetlov I’ve been working on an adapter for aiohttp, and I'm currently following a similar pattern with regards to setting up configuration before setting up the primary application component:
https://github.com/JustinTArthur/serverwamp#example-wamp-microservice
I’m curious though why you chose that pattern, vs something like Flask where the app object is constructed first thing, and handlers are registered to it afterwards. Seems fine, am just curious
https://github.com/JustinTArthur/serverwamp#example-wamp-microservice
I’m curious though why you chose that pattern, vs something like Flask where the app object is constructed first thing, and handlers are registered to it afterwards. Seems fine, am just curious
In aiohttp
Application is statefull object: user data can be stored in app, e.g. app['database'] = db. So, applications can be instantiated in user code somewhere in main() function.
In Flask application is recommended to be used as singleton available on import time to make @app.route('/') work. Basically, it means that Flask really encourages the only app per process.
There is no good space for saving user data, module-level singletons very often preferred over-complicated [flask.g] (https://flask.palletsprojects.com/en/1.1.x/appcontext/) application context.
Testing is also complicated: since massive global variable usage (explicit or implicit by flask contexts) tests isolation can be broken easily in invisible way.
Flask provides approaches to handle these things, the framework has had proven it's popularity and maturity.
In aiohttp design, we strived to avoid mutable global states at all. We have proved that HTTP framework can be constructed without any global variable. You can use them in your code, sure -- contextvars are handy sometimes. But aiohttp doesn't enforce global states itself.
@dkliban we use Travis CI for automated deployment initiated by git tag.
Currently, it fails like this: https://travis-ci.com/aio-libs/aiohttp/jobs/224450594 I'm working slowly to fix these spikes. 3 days ago we had much more build errors.
Currently, it fails like this: https://travis-ci.com/aio-libs/aiohttp/jobs/224450594 I'm working slowly to fix these spikes. 3 days ago we had much more build errors.
(possibly relevant here: pytest-dev/pytest#5751)
I really want aio-libs/aiohttp@167fe9f
if so you have missed the
make cythonize step
master builds for me
you should use one
For example, now aiohttp has two calls per HTTP message: one for headers and another one for HTTP body. We do it for multiple reasons.
Under heavy load two
In turn, for single
write()call per message (whatever protocol you use it has message concept most likely) if asyncio buffer is disabled.For example, now aiohttp has two calls per HTTP message: one for headers and another one for HTTP body. We do it for multiple reasons.
Under heavy load two
write() calls with disabled buffering is slower than buffered mode about 30% if I recall numbers correctly.In turn, for single
write() call the bufferless mode is faster.
Something like
writev()can solve multiple buffers problem without overhead, asyncio doesn't support it yet.
so I'd expect a single
write to be the right thing to do, even with buffering enabled, to get header+body into a single packet
and I wouldn't expect the send buffer to matter unless you're experiencing backpressure, which is exactly the case where you don't want the buffer because it increases latency...
The speed difference was observed under heavy load only. Maybe in this mode hysteresis provided by asyncio (low and high watermarks for output buffer) significantly reduced the number of syscalls but did
For HTTP messages are tiny often, several messages can be merged and sent by single syscall if buffering is on.
buffer.extend() (mem copy should be cheaper)?For HTTP messages are tiny often, several messages can be merged and sent by single syscall if buffering is on.
@asvetlov but unless you're doing pipelining, you can't merge more than a single headers+body together, and pipelining isn't relevant to real-world usage, so I don't think the write buffer can outperform calling
write once bufferless
....oh, and that's what you already said you observed in benchmarks, up above
Hello!
What is the best way to properly handle requests like:
GET localhost:8080//..%5Cetc/password
HTTP/1.1 500 Internal Server Error
Server: Python/3.7 aiohttp/3.5.4
500 Internal Server Error
Server got itself in troubleError handling request
Traceback (most recent call last):
File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 659, in _encode_host
ip = ip_address(ip)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ipaddress.py", line 54, in ip_address
address)
ValueError: '..' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 662, in _encode_host
host = idna.encode(host, uts46=True).decode("ascii")
File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 358, in encode
s = alabel(label)
File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 270, in alabel
ulabel(label)
File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 304, in ulabel
check_label(label)
File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 239, in check_label
raise IDNAError('Empty Label')
idna.core.IDNAError: Empty Label
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/encodings/idna.py", line 165, in encode
raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 105, in impl
request, path)
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 22, in _check_request_resolves
alt_request = request.clone(rel_url=path)
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_request.py", line 709, in clone
remote=remote)
File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_request.py", line 168, in clone
new_url = URL(rel_url)
File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 168, in __new__
val.username, val.password, host, port, encode=True
File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 676, in _make_netloc
ret = cls._encode_host(host)
File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 664, in _encode_host
host = host.encode("idna").decode("ascii")
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Hi, I've written something that uses an aiohttp client internally, as part of a long running task (it's a graph that lazy loads vertices from an upstream API).
I'd like to use this in a jupyter notebook, but I'm not sure how to manage the ClientSession anymore, because it's closed once a lexical async with scope is exited (I don't want to create a new session for each cell)
@nothingmuch this is a decision I’m having to make a lot lately. I think aiohttp is trying to move away from the older usage pattern. I’ve occasionally gotten it to work in a jam, but more and more I’m creating instances for my long running tasks and bootstrapping them with any connections/contexts/managers they’d need like this:
async with aiohttp.ClientSession() as http_session:
long_running_thing = LongRunningThing(http_session)
await long_running_thing.do_things_for_a_long_time()
That pattern also makes it easier to unit test, say with a mock http client.
yeah i was thinking of doing that, the thing in question is a sparse, lazily fetched graph where vertices are basically just objects with awaitable attrs, and tasks are created in getattr to fetch the data. maybe that's a bad design (i'm new to python and on a deadline), but it seemed like the only way to get this to work with a coroutine in the background with a session context was to implement some sort of queue mechanism, which i was hoping to avoid
for now i think i'm happy with using it the old way just in jupyter, just to avoid the distracting boilerplate
if i run into trouble i can avoid async entirely by just relying on the cached copy (the db is sqlite so blocking anyway, the only async code is for lazy downloading of missing data)
hi guys, any plans to release new version of aioredis soon? (looking forward to aio-libs/aioredis#625)