by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jan 31 2019 19:45
    dcecile commented #503
  • Jan 31 2019 19:40
    blueyed commented #526
  • Jan 31 2019 13:29
    codecov[bot] commented #526
  • Jan 31 2019 13:26
    codecov[bot] commented #526
  • Jan 31 2019 11:59
    codecov[bot] commented #526
  • Jan 31 2019 11:59
    popravich synchronize #526
  • Jan 31 2019 11:59

    popravich on travis_pypy

    .travis.yml: chaching pip packa… Makefile: build/install only re… .travis.yml: cache redis-server… (compare)

  • Jan 31 2019 11:44
    codecov[bot] commented #526
  • Jan 31 2019 11:36
    codecov[bot] commented #526
  • Jan 31 2019 11:35
    codecov[bot] commented #526
  • Jan 31 2019 11:34
    codecov[bot] commented #526
  • Jan 31 2019 11:24
    codecov[bot] commented #526
  • Jan 31 2019 11:23
    codecov[bot] commented #526
  • Jan 31 2019 10:04
    gyermolenko commented #503
  • Jan 31 2019 09:31
    gyermolenko commented #431
  • Jan 31 2019 09:23
    Natim closed #444
  • Jan 31 2019 09:23
    Natim commented #444
  • Jan 31 2019 09:09
    gyermolenko commented #444
  • Jan 31 2019 05:32
    codecov[bot] commented #539
  • Jan 31 2019 05:32
    vir-mir synchronize #539
藤原太一殺す
@idontforgivefujiwara_gitlab
Hi, I'm trying to write some middleware for a websockets connection under aiohttp. Given the request how can I read the message / payload? The documentation only shows doing so in the context of giving a response, but I don't want to give a response, I want that to be handled as it would have been.
Justin Turner Arthur
@JustinTArthur
@idontforgivefujiwara_gitlab WebSocketResponse is not a response to a WebSocket message but to the original HTTP request. It is basically an HTTP response that includes the entire lifecycle of the WebSocket communication happening within the HTTP response. In order to be able to read a message/payload like you want, you will have to at least start the WebSocketResponse and call .prepare(request) on it. This would need to happen in your middleware if that’s where you want to start processing WebSocket messages. You can pass the WebSocketResponse you’ve started down the chain, e.g. by including it in the request object, e.g.
# In middleware…
ws = web.WebSocketResponse()
ws.prepare(request)

# (Here is where you might read a message or two)

# ensuring it is passed to the eventual handler…
request['ws'] = ws   # handler should also return this object
resp = await handler(request)
return resp
藤原太一殺す
@idontforgivefujiwara_gitlab

@JustinTArthur Thanks for the response. When I do:

 ws = web.WebSocketResponse()
 await ws.prepare(request)
 async for msg in ws:
         logger.debug(f'msg: {msg} | {dir(msg)}')
 request['ws'] = ws

I get "Client protocols ['graphql-ws'] don’t overlap server-known ones ()".

Removing the loop gives me " File "/root/.local/share/virtualenvs/project-XkHUY7Ky/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 257, in set_parser
assert self._payload_parser is None".

Then removing the await gives me the resultset I'd expect though obviously the middleware's not doing anything.

Justin Turner Arthur
@JustinTArthur
If your client requires certain protocols, you can configure the server to hint what it allows. e.g.
ws = web.WebSocketResponse(protocols=(‘graphql-ws')) and don’t forget that the async for loop will keep iterating until the WebSocket is closed, so if you want to continue past your middleware prior to close, you’ll need to break out of the loop. I don’t remember if that closes the connection. if so, you’ll need to pass the iterator instead of the ws itself.
藤原太一殺す
@idontforgivefujiwara_gitlab
@JustinTArthur perfect, protocol is exactly what it was. The looping would cause no results to be returned, right? If so I'm getting that, break gives me the self._payload_parser is None exception again. I tried setting request['ws'] to msg but that didn't change anything. Is there anywhere this is documented rather than bugging you? I don't want to impose.
thanks for the help so far by the way. I'd have been lost without it
Justin Turner Arthur
@JustinTArthur
Reading messages from WebSockets in middleware is somewhat unusual, so I don’t think there’ll be much in the way of docs on it, but instead of iterating it in the middleware, you might be able to just do ws.receive_str() for however many messages you want to process, then maybe a downstream handler will be able to iterate the ws with async for.
藤原太一殺す
@idontforgivefujiwara_gitlab
Yeah, I figured as much. Really I just want to get the connection_init message because that's where the apollo graphql connectionParams values are passed. There's zero documentation about any of this on their side either. Seems like uncharted territory :) I'll try your suggestion. thanks
藤原太一殺す
@idontforgivefujiwara_gitlab
so I'm getting the data you said but still getting a payload parser exception :/ So close. I'm wondering if that's why they have "Note: Can only be called by the request handling task." in the docs. https://docs.aiohttp.org/en/stable/web_reference.html?highlight=receive_str#aiohttp.web.WebSocketResponse.receive_str
euri10
@euri10
Why 8 to y55 like
shaqarava
@shaqarava
Why this now?
DeprecationWarning: Setting custom web.Application.db_engine attribute is discouraged app.db_engine = db_engine
I must have missed some change
Andrey Oleynik
@oleynikandrey
Hello, sometimes I see invocation of on_request_exception withoutaiohttp.TraceRequestExceptionParams.exception when it could happen and how to find a reason?
Link Mauve
@linkmauve_gitlab
Hi, I’d like to build multidict for a Flatpak application, here is what it ends up with, despite the org.freedesktop.Sdk environment having pip 19.2.3:
Running: pip3 install --prefix=/app .
Processing /run/build/python3-multidict
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7vxxn82q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel
       cwd: None
  Complete output (8 lines):
  Collecting pip>=18
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e609c3d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60a5c50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60a5e50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60c5050>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60c5210>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
    ERROR: Could not find a version that satisfies the requirement pip>=18 (from versions: none)
  ERROR: No matching distribution found for pip>=18
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7vxxn82q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.
How can I tell it that pip is already installed and usable, and that it can skip downloading this dependency?
Link Mauve
@linkmauve_gitlab
Alternatively, could you generate wheels for ARMv7 and AArch64 so that I wouldn’t have to build it from source?
What is pyproject.toml?
If I remove it I can build successfully, but this seems like a very brittle solution.
Hugo Almeida
@openalmeida
Hello, anyone tell me about aiohttp's or/and requests3's file related asynchronous IO model ?
how they read file asynchronously, there are methods about that,
such as multi-thread based/multi-process based/epoll based/linux aio based ...
Nathaniel J. Smith
@njsmith
@openalmeida aiohttp uses loop.run_in_executor, which uses a thread pool
Hugo Almeida
@openalmeida
thanks a lot ! got it.
Nathaniel J. Smith
@njsmith
that's kind of the only viable way to support async disk I/O currently. The io_uring project for the Linux kernel is looking promising to maybe provide another option at some point, but it's not really ready yet, and none of the asyncio stack is set up to use it.
and requests3 doesn't exist.
Hugo Almeida
@openalmeida
@linkmauve_gitlab 1st, pyproject.toml is the PEP 518 or the PEP 517 related. 2ed, type which pip-3 will result pip installed or not and pip version should new enough to support pyproject.toml which configure pip stuff.
@njsmith yes, i came from liburing's python binding (io_uring based).
@njsmith requests 3 is WIP. https://requests.readthedocs.io/projects/3/
Nathaniel J. Smith
@njsmith
well, that's what Kenneth says, though I don't believe him. but in any case, even if it might exist someday, it doesn't exist now.
Hugo Almeida
@openalmeida
yes, wip always meaning none.
Justin Turner Arthur
@JustinTArthur
@shaqarava, the new way is to just set an arbitrary item (instead of attribute) on the application object. e.g. app['db'] = db_engine instead of app.db_engine = db_engine.
Andrey Oleynik
@oleynikandrey

Hello, sometimes I see invocation of on_request_exception withoutaiohttp.TraceRequestExceptionParams.exception when it could happen and how to find a reason?

No thoughts?

Oleg Korsak
@kamikaze

hey... I'm having issues with running aiohttp:

async def main():
    await pg.init(dsn=settings.DB_DSN)
    await _bootstrap_db()

asyncio.run(main())

web.run_app(app, host=settings.HTTP_ADDRESS, port=settings.HTTP_PORT)

and getting:
RuntimeError: There is no current event loop in thread 'MainThread'.

I can get it working with get_event_loop and running main within it... but then I'm getting DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
Justin Turner Arthur
@JustinTArthur
Both asyncio.run and web.run_app start an event loop and take it through its entire lifecycle. You can either implement your own runner and do the things web.run_app does or use something like app.cleanup_ctx to initialize and bootstrap your database during aiohttp app run.
Joongi Kim
@achimnol
@asvetlov um... when are the aiohttp middleware invoked: after reading the request body or before reading the request body but after reading the request header?
is there any way to control this?
for example, i have auth_required middleware to check the headers for valid credentials, and it should return the "ok" or "not ok" response before accepting a large file upload
currently my server seems to just read all the body upload and then replies back with "invalid credential"
Joongi Kim
@achimnol
of course, due to the limitation of HTTP, we cannot forbid the client to send some forehead of the large upload through the network,
but at least we could forcibly cancel the request handling once we asynchronously detect the header credential is not valid,
saving a great amount of network traffic though not completely
Andrew Svetlov
@asvetlov

@achimnol
@asvetlov um... when are the aiohttp middleware invoked: after reading the request body or before reading the request body but after reading the request header?
is there any way to control this?

The second (headers are available, the body is not read).
Ideally, a middleware should not read the body if it wants to pass the control to the next handler in the middleware chain.

It can work in some circumstances because aiohttp request caches the read body bytes and returns it from read() or text() calls but streaming API (request.content.*) is not compatible with this caching.

Oleg Korsak
@kamikaze

Both asyncio.run and web.run_app start an event loop and take it through its entire lifecycle. You can either implement your own runner and do the things web.run_app does or use something like app.cleanup_ctx to initialize and bootstrap your database during aiohttp app run.

thanks, that worked :)

Justin Turner Arthur
@JustinTArthur
Awesome :)
ioon403
@ioon403
Привет, ребята. Только начал проникаться асинхронным пайтоном. Вопрос : есть ли возможность выполнять паб/саб запросы в редис из асинхронной функции не дожидаясь ее полного выполнения? Кейс : есть долгая операция разбитая на кучу мелких корутин. В беграунде висит таска которая слушает канал и при появлении сообщения оповещает всех кто подключен по вебсокетам. Пытаюсь паблишить при выполнении тех или иных задач, чтобы подключенные клиенты могли запускать свои задачи получив сообщения. В результате всё паблишится лишь по завершении общего пайплайна. Направьте пожалуйста мучил треды как умел, пихал туда паб, отдельно ставил синхронный редис создавал подключение и паблишил им - пока все безрезультатно (паблишится лишь при завершении всех операций)
ioon403
@ioon403
Eng.version : Hi, guys. I just started getting into asynchronous Python. Question:is it possible to execute pub / sub queries in redis from an asynchronous function without waiting for it to complete? Case : there is a long operation broken down into a bunch of small coroutines. In the background, there is a task that listens to the redis channel and notifies everyone who is connected via websockets when a message appears. I try to publish when performing small tasks, so that connected clients can run their own tasks after receiving messages. As a result, everything is published only at the end of the general pipeline. Direct me please. Tormented the threads as i could, shoved the pub there, separately intalled a synchronous redis, created a connection and published it - so far everything is to no avail (messages published only at the end of all operations)
Link Mauve
@linkmauve_gitlab
@openalmeida thanks for your reply, but I don’t know how to type a command while building a flatpak, I’ll try a few things.
For now I’ve removed the pyproject.toml file since it lets multidict build, but that’s obviously a bad solution.
ioon403
@ioon403
It's ok. I figured it out myself =)
Oleg Korsak
@kamikaze
pip install aiohttp[speedups] pulls brotlipy, while I see that in 2019 there was a switch from it to brotli. Am I missing something?
Justin Turner Arthur
@JustinTArthur
@asvetlov I’m assuming you’re pretty busy these days. Anything we can do to help you get 3.7 out the door?