https://github.com/aio-libs The set of asyncio-based libraries built with high quality
Hi, I'm trying to write some middleware for a websockets connection under aiohttp. Given the request how can I read the message / payload? The documentation only shows doing so in the context of giving a response, but I don't want to give a response, I want that to be handled as it would have been.
@idontforgivefujiwara_gitlab
WebSocketResponse is not a response to a WebSocket message but to the original HTTP request. It is basically an HTTP response that includes the entire lifecycle of the WebSocket communication happening within the HTTP response. In order to be able to read a message/payload like you want, you will have to at least start the WebSocketResponse and call .prepare(request) on it. This would need to happen in your middleware if that’s where you want to start processing WebSocket messages. You can pass the WebSocketResponse you’ve started down the chain, e.g. by including it in the request object, e.g.# In middleware…
ws = web.WebSocketResponse()
ws.prepare(request)
# (Here is where you might read a message or two)
# ensuring it is passed to the eventual handler…
request['ws'] = ws # handler should also return this object
resp = await handler(request)
return resp@JustinTArthur Thanks for the response. When I do:
ws = web.WebSocketResponse()
await ws.prepare(request)
async for msg in ws:
logger.debug(f'msg: {msg} | {dir(msg)}')
request['ws'] = wsI get "Client protocols ['graphql-ws'] don’t overlap server-known ones ()".
Removing the loop gives me " File "/root/.local/share/virtualenvs/project-XkHUY7Ky/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 257, in set_parser
assert self._payload_parser is None".
Then removing the await gives me the resultset I'd expect though obviously the middleware's not doing anything.
If your client requires certain protocols, you can configure the server to hint what it allows. e.g.
ws = web.WebSocketResponse(protocols=(‘graphql-ws')) and don’t forget that the async for loop will keep iterating until the WebSocket is closed, so if you want to continue past your middleware prior to close, you’ll need to break out of the loop. I don’t remember if that closes the connection. if so, you’ll need to pass the iterator instead of the ws itself.
@JustinTArthur perfect, protocol is exactly what it was. The looping would cause no results to be returned, right? If so I'm getting that, break gives me the self._payload_parser is None exception again. I tried setting request['ws'] to msg but that didn't change anything. Is there anywhere this is documented rather than bugging you? I don't want to impose.
thanks for the help so far by the way. I'd have been lost without it
Reading messages from WebSockets in middleware is somewhat unusual, so I don’t think there’ll be much in the way of docs on it, but instead of iterating it in the middleware, you might be able to just do
ws.receive_str() for however many messages you want to process, then maybe a downstream handler will be able to iterate the ws with async for.
Yeah, I figured as much. Really I just want to get the connection_init message because that's where the apollo graphql connectionParams values are passed. There's zero documentation about any of this on their side either. Seems like uncharted territory :) I'll try your suggestion. thanks
so I'm getting the data you said but still getting a payload parser exception :/ So close. I'm wondering if that's why they have "Note: Can only be called by the request handling task." in the docs. https://docs.aiohttp.org/en/stable/web_reference.html?highlight=receive_str#aiohttp.web.WebSocketResponse.receive_str
I must have missed some change
Hi, I’d like to build multidict for a Flatpak application, here is what it ends up with, despite the org.freedesktop.Sdk environment having pip 19.2.3:
Running: pip3 install --prefix=/app .
Processing /run/build/python3-multidict
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7vxxn82q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel
cwd: None
Complete output (8 lines):
Collecting pip>=18
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e609c3d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60a5c50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60a5e50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60c5050>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f05e60c5210>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pip/
ERROR: Could not find a version that satisfies the requirement pip>=18 (from versions: none)
ERROR: No matching distribution found for pip>=18
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7vxxn82q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pip>=18' 'setuptools>=40' wheel Check the logs for full command output.
How can I tell it that pip is already installed and usable, and that it can skip downloading this dependency?
What is pyproject.toml?
If I remove it I can build successfully, but this seems like a very brittle solution.
and requests3 doesn't exist.
@njsmith yes, i came from liburing's python binding (io_uring based).
@njsmith requests 3 is WIP. https://requests.readthedocs.io/projects/3/
hey... I'm having issues with running aiohttp:
async def main():
await pg.init(dsn=settings.DB_DSN)
await _bootstrap_db()
asyncio.run(main())
web.run_app(app, host=settings.HTTP_ADDRESS, port=settings.HTTP_PORT)and getting:RuntimeError: There is no current event loop in thread 'MainThread'.
I can get it working with get_event_loop and running main within it... but then I'm getting
DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
Both
asyncio.run and web.run_app start an event loop and take it through its entire lifecycle. You can either implement your own runner and do the things web.run_app does or use something like app.cleanup_ctx to initialize and bootstrap your database during aiohttp app run.
is there any way to control this?
for example, i have
auth_required middleware to check the headers for valid credentials, and it should return the "ok" or "not ok" response before accepting a large file upload
currently my server seems to just read all the body upload and then replies back with "invalid credential"
but at least we could forcibly cancel the request handling once we asynchronously detect the header credential is not valid,
saving a great amount of network traffic though not completely
@achimnol
@asvetlov um... when are the aiohttp middleware invoked: after reading the request body or before reading the request body but after reading the request header?
is there any way to control this?
The second (headers are available, the body is not read).
Ideally, a middleware should not read the body if it wants to pass the control to the next handler in the middleware chain.
It can work in some circumstances because aiohttp request caches the read body bytes and returns it from read() or text() calls but streaming API (request.content.*) is not compatible with this caching.
Both
asyncio.runandweb.run_appstart an event loop and take it through its entire lifecycle. You can either implement your own runner and do the thingsweb.run_appdoes or use something likeapp.cleanup_ctxto initialize and bootstrap your database during aiohttp app run.
thanks, that worked :)
Привет, ребята. Только начал проникаться асинхронным пайтоном. Вопрос : есть ли возможность выполнять паб/саб запросы в редис из асинхронной функции не дожидаясь ее полного выполнения? Кейс : есть долгая операция разбитая на кучу мелких корутин. В беграунде висит таска которая слушает канал и при появлении сообщения оповещает всех кто подключен по вебсокетам. Пытаюсь паблишить при выполнении тех или иных задач, чтобы подключенные клиенты могли запускать свои задачи получив сообщения. В результате всё паблишится лишь по завершении общего пайплайна. Направьте пожалуйста мучил треды как умел, пихал туда паб, отдельно ставил синхронный редис создавал подключение и паблишил им - пока все безрезультатно (паблишится лишь при завершении всех операций)
Eng.version : Hi, guys. I just started getting into asynchronous Python. Question:is it possible to execute pub / sub queries in redis from an asynchronous function without waiting for it to complete? Case : there is a long operation broken down into a bunch of small coroutines. In the background, there is a task that listens to the redis channel and notifies everyone who is connected via websockets when a message appears. I try to publish when performing small tasks, so that connected clients can run their own tasks after receiving messages. As a result, everything is published only at the end of the general pipeline. Direct me please. Tormented the threads as i could, shoved the pub there, separately intalled a synchronous redis, created a connection and published it - so far everything is to no avail (messages published only at the end of all operations)
For now I’ve removed the pyproject.toml file since it lets multidict build, but that’s obviously a bad solution.