Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 19:45
    dcecile commented #503
  • Jan 31 19:40
    blueyed commented #526
  • Jan 31 13:29
    codecov[bot] commented #526
  • Jan 31 13:26
    codecov[bot] commented #526
  • Jan 31 11:59
    codecov[bot] commented #526
  • Jan 31 11:59
    popravich synchronize #526
  • Jan 31 11:59

    popravich on travis_pypy

    .travis.yml: chaching pip packa… Makefile: build/install only re… .travis.yml: cache redis-server… (compare)

  • Jan 31 11:44
    codecov[bot] commented #526
  • Jan 31 11:36
    codecov[bot] commented #526
  • Jan 31 11:35
    codecov[bot] commented #526
  • Jan 31 11:34
    codecov[bot] commented #526
  • Jan 31 11:24
    codecov[bot] commented #526
  • Jan 31 11:23
    codecov[bot] commented #526
  • Jan 31 10:04
    gyermolenko commented #503
  • Jan 31 09:31
    gyermolenko commented #431
  • Jan 31 09:23
    Natim closed #444
  • Jan 31 09:23
    Natim commented #444
  • Jan 31 09:09
    gyermolenko commented #444
  • Jan 31 05:32
    codecov[bot] commented #539
  • Jan 31 05:32
    vir-mir synchronize #539
Paul Rudin
@PaulRudin
if it's just for a single view why not just put the logic in the handler for that view?
Dennis Kliban
@dkliban
Could you take a look at this small bugfix @asvetlov ? aio-libs/aiohttp#3960
Justin Turner Arthur
@JustinTArthur
@asvetlov I’ve been working on an adapter for aiohttp, and I'm currently following a similar pattern with regards to setting up configuration before setting up the primary application component:
https://github.com/JustinTArthur/serverwamp#example-wamp-microservice
I’m curious though why you chose that pattern, vs something like Flask where the app object is constructed first thing, and handlers are registered to it afterwards. Seems fine, am just curious
Dennis Kliban
@dkliban
@asvetlov Thank you for merging my PR. what is the plan for releasing another version of aiohttp? looks like there has not been a release since January
Andrew Svetlov
@asvetlov
@JustinTArthur thanks for the good question.
In aiohttp Application is statefull object: user data can be stored in app, e.g. app['database'] = db. So, applications can be instantiated in user code somewhere in main() function.
Andrew Svetlov
@asvetlov

In Flask application is recommended to be used as singleton available on import time to make @app.route('/') work. Basically, it means that Flask really encourages the only app per process.
There is no good space for saving user data, module-level singletons very often preferred over-complicated [flask.g] (https://flask.palletsprojects.com/en/1.1.x/appcontext/) application context.
Testing is also complicated: since massive global variable usage (explicit or implicit by flask contexts) tests isolation can be broken easily in invisible way.

Flask provides approaches to handle these things, the framework has had proven it's popularity and maturity.
In aiohttp design, we strived to avoid mutable global states at all. We have proved that HTTP framework can be constructed without any global variable. You can use them in your code, sure -- contextvars are handy sometimes. But aiohttp doesn't enforce global states itself.

Andrew Svetlov
@asvetlov
@dkliban there are few things to be done before making a release. The broken build system is a part of this task :)
Dennis Kliban
@dkliban
@asvetlov Thank you for getting back to me. What build system do you use? Is this actively being worked on? Is there any way I can be of help?
Justin Turner Arthur
@JustinTArthur
@asvetlov thanks for the explanations on the design pattern of app/configuration!
Andrew Svetlov
@asvetlov
@JustinTArthur sure!
@dkliban we use Travis CI for automated deployment initiated by git tag.
Currently, it fails like this: https://travis-ci.com/aio-libs/aiohttp/jobs/224450594 I'm working slowly to fix these spikes. 3 days ago we had much more build errors.
Nathaniel J. Smith
@njsmith
@asvetlov do you remember why you set 3.5.3 as the minimum aiohttp version? I know about a bunch of changes in 3.5.1->3.5.2, but I'm not sure what changed in 3.5.2->3.5.3
(possibly relevant here: pytest-dev/pytest#5751)
Andrew Svetlov
@asvetlov
@njsmith yes, sure. In 3.5.2 and above asyncio.get_event_loop() returns a loop installed by set_event_loop() instead a real loop used for running async code, e.g. loop.run_until_complete(coro()).
It makes a mess easily, the error is hard detectable by newbies.
Nathaniel J. Smith
@njsmith
@asvetlov ahhh right 3.5.3 is where the new running loop tracking stuff landed. Thanks!
YoilyL
@YoilyL
when is the next aiohttp release scheduled? I want to install from master but the *.c files are missing so setup fails
I really want aio-libs/aiohttp@167fe9f
Bryan Kok
@Transfusion
@YoilyL did you experience something like x86_64-linux-gnu-gcc: error: aiohttp/_websocket.c: No such file or directory x86_64-linux-gnu-gcc: fatal error: no input files?
if so you have missed the make cythonize step
master builds for me
YoilyL
@YoilyL
oh. will try. where is this step documented?
YoilyL
@YoilyL
worked.. thanks a million!
YoilyL
@YoilyL
except that now my logs are full of web_protocol.py:380 ERROR: Error handling request without a traceback
YoilyL
@YoilyL
probably my timeout errors.. disregard
Justin Turner Arthur
@JustinTArthur
Can any of you think of any compelling reasons to ever use the asyncio network send buffer except for emergencies? I’m thinking of setting the upper/lower watermarks to 0 on all my asyncio projects and just use the OS’ buffers. Am I missing something?
Nathaniel J. Smith
@njsmith
@JustinTArthur AFAICT that's generally what you want yeah
Justin Turner Arthur
@JustinTArthur
Thanks, @njsmith. I was looking around, and it seems like a lot of async frameworks do it because “that’s what node.js or some other project does"
Andrew Svetlov
@asvetlov
you should use one write()call per message (whatever protocol you use it has message concept most likely) if asyncio buffer is disabled.
For example, now aiohttp has two calls per HTTP message: one for headers and another one for HTTP body. We do it for multiple reasons.
Under heavy load two write() calls with disabled buffering is slower than buffered mode about 30% if I recall numbers correctly.
In turn, for single write() call the bufferless mode is faster.
Something like writev()can solve multiple buffers problem without overhead, asyncio doesn't support it yet.
Nathaniel J. Smith
@njsmith
@asvetlov huh, why does the buffer speed things up? I thought that asyncio's write always attempted to push the data into the kernel immediately, regardless of the buffer, and asyncio's buffer only came into play if the kernel's send buffer was filled up
so I'd expect a single write to be the right thing to do, even with buffering enabled, to get header+body into a single packet
and I wouldn't expect the send buffer to matter unless you're experiencing backpressure, which is exactly the case where you don't want the buffer because it increases latency...
Andrew Svetlov
@asvetlov
The speed difference was observed under heavy load only. Maybe in this mode hysteresis provided by asyncio (low and high watermarks for output buffer) significantly reduced the number of syscalls but did buffer.extend() (mem copy should be cheaper)?
For HTTP messages are tiny often, several messages can be merged and sent by single syscall if buffering is on.
Nathaniel J. Smith
@njsmith
@asvetlov hmm, if you're sending headers + bodies separately then yeah, it basically ends up implementing the "copy headers and bodies into a single buffer and call write once" strategy
@asvetlov but unless you're doing pipelining, you can't merge more than a single headers+body together, and pipelining isn't relevant to real-world usage, so I don't think the write buffer can outperform calling write once bufferless
....oh, and that's what you already said you observed in benchmarks, up above
Justin Turner Arthur
@JustinTArthur
Interesting
Andrew Svetlov
@asvetlov
to the note: I didn't use HTTP pipelining. It is useless in real world, I agree.
Andrey Oleynik
@oleynikandrey

Hello!

What is the best way to properly handle requests like:

GET localhost:8080//..%5Cetc/password

HTTP/1.1 500 Internal Server Error
Server: Python/3.7 aiohttp/3.5.4

500 Internal Server Error

Server got itself in trouble
Error handling request
Traceback (most recent call last):
  File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 659, in _encode_host
    ip = ip_address(ip)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ipaddress.py", line 54, in ip_address
    address)
ValueError: '..' does not appear to be an IPv4 or IPv6 address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 662, in _encode_host
    host = idna.encode(host, uts46=True).decode("ascii")
  File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 358, in encode
    s = alabel(label)
  File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 270, in alabel
    ulabel(label)
  File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 304, in ulabel
    check_label(label)
  File "/Users/user/project/venv/lib/python3.7/site-packages/idna/core.py", line 239, in check_label
    raise IDNAError('Empty Label')
idna.core.IDNAError: Empty Label

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 105, in impl
    request, path)
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 22, in _check_request_resolves
    alt_request = request.clone(rel_url=path)
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_request.py", line 709, in clone
    remote=remote)
  File "/Users/user/project/venv/lib/python3.7/site-packages/aiohttp/web_request.py", line 168, in clone
    new_url = URL(rel_url)
  File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 168, in __new__
    val.username, val.password, host, port, encode=True
  File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 676, in _make_netloc
    ret = cls._encode_host(host)
  File "/Users/user/project/venv/lib/python3.7/site-packages/yarl/__init__.py", line 664, in _encode_host
    host = host.encode("idna").decode("ascii")
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Justin Turner Arthur
@JustinTArthur
@oleynikandrey a 500 error seems like a reasonable response for that, though it is quite a lot of log noise. If you’re looking at bigger picture solutions, a WAF can often filter traffic like that out before it hits your webapp.
Yuval Kogman
@nothingmuch

Hi, I've written something that uses an aiohttp client internally, as part of a long running task (it's a graph that lazy loads vertices from an upstream API).

I'd like to use this in a jupyter notebook, but I'm not sure how to manage the ClientSession anymore, because it's closed once a lexical async with scope is exited (I don't want to create a new session for each cell)

Yuval Kogman
@nothingmuch
hmm, i think i misunderstood the async with usage as being mandatory somehow
Justin Turner Arthur
@JustinTArthur
@nothingmuch this is a decision I’m having to make a lot lately. I think aiohttp is trying to move away from the older usage pattern. I’ve occasionally gotten it to work in a jam, but more and more I’m creating instances for my long running tasks and bootstrapping them with any connections/contexts/managers they’d need like this:
async with aiohttp.ClientSession() as http_session:
    long_running_thing = LongRunningThing(http_session)
    await long_running_thing.do_things_for_a_long_time()
That pattern also makes it easier to unit test, say with a mock http client.
Yuval Kogman
@nothingmuch
yeah i was thinking of doing that, the thing in question is a sparse, lazily fetched graph where vertices are basically just objects with awaitable attrs, and tasks are created in getattr to fetch the data. maybe that's a bad design (i'm new to python and on a deadline), but it seemed like the only way to get this to work with a coroutine in the background with a session context was to implement some sort of queue mechanism, which i was hoping to avoid
for now i think i'm happy with using it the old way just in jupyter, just to avoid the distracting boilerplate
if i run into trouble i can avoid async entirely by just relying on the cached copy (the db is sqlite so blocking anyway, the only async code is for lazy downloading of missing data)
fegonfe
@fegonfe
hi @asvetlov , janus latest package that works for arm is janus-0.3.2-py3-none-any.whl. Any ideas about 0.40?
unintended
@unintended
hi guys, any plans to release new version of aioredis soon? (looking forward to aio-libs/aioredis#625)