I’ve been running into this issue quite a bit recently.
AWS Batch Error: CannotCreateContainerError: Error response from daemon: devmapper: Thin Pool has 4115 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior This could be a transient error. Use @retry to retry.Removing any EC2s controlled by Metaflow and letting the ASG create new ones seems to temporarily solve the problem but it keeps reappearing. Any advice on how to mitigate?
5 replies
1 reply
python flow.py run causes the machine I'm running this on to go OOM after launching a Python process for each job. This is after setting --max-workers and --max-num-splits high enough to allow this many jobs.
11 replies
5 replies
I got this very weird issue running metaflow locally with a step that import pandas... while running the code out of metaflow it all works fine... but the same code in the same env with metaflow I get
ImportError: cannot import name 'Collection' from 'typing' (/home/hexa/.cache/pypoetry/virtualenvs/nima-images-H6fn72k1-py3.7/lib/python3.7/site-packages/typing.py)Ideas ?
5 replies
Key lessons to date:
- Using docker images is a lot less trouble then conda.. or to try to use pip via a custom decorator
- Using patch_env is great to get debugging to work with session based aws keys
- Dependency clashes due to metaflow actually being part of your app, can be really painful
- Setting a minimum CPU to the cluster can leave with with a GPU instance running instead of pure CPU....
- Using step functions for long running processes is still a bit painful, can't wait for improvements there :)
5 replies
Hey Metaflow, one unexpected discovery I've stumbled upon is the orchestration of the DAG is performed locally, even when running on AWS Batch.
Additionally, I created an access list for our AWS API Gateway that our Metaflow API uses, as one layer of security.
This means that we have to leave our data scientists machines running and on the VPN, during long running training flows.
Is there any configuration of Metaflow to allow the orchestration to be performed remotely and allow our development machines to be disconnected from the VPN or even shut down during execution?
1 reply
Has anyone run into intermmitent errors from the Metaflow service?
failed (code 500): {“message”: “Internal server error”}This seems to happen on flows with multiple steps where the pipeline starts fine and runs a few steps, but then fails with the above error
17 replies
7 replies
/migration_service/migration_server.py:17: DeprecationWarning: loop argument is deprecated
app = web.Application(loop=loop)
/migration_service/migration_server.py:28: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
AttributeError: 'NoneType' object has no attribute 'cursor'
/bin/sh: 1: metadata_service: not found
@batch(image='custom-image') . What would be the best way to build on top of the default image to ensure requirements for Metaflow are preserved? How does Metaflow install necessary dependencies on the default image?
4 replies
curl is not installed in the image. Seems to be used to lookup the dynamodb host. FWIW not an unreasonable dependency, and I was surprised to find that curlwasn't already baked into the continuumio/miniconda3:latest image. Once installing it in the docker image the dynamodb host resolution worked as expected.
5 replies
Hello all, I'm trying to take a look at the artifacts associated with one of our SFN executions but when trying to call:
Step(...).task.dataWe run into the following error:
ServiceException: Metadata request (/flows/{Flow}/runs/{Run}/steps/start/tasks/{Task}/artifacts) failed (code 500): 500 Internal Server ErrorLooking at the logs from our metadata service, I see the following error:
Traceback (most recent call last):
File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 140, in get_artifacts_by_task
artifacts.body)
File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 355, in _filter_artifacts_by_attempt_id
attempt_id = ArtificatsApi._get_latest_attempt_id(artifacts)
File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 349, in _get_latest_attempt_id
if artifact['attempt_id'] > attempt_id:
TypeError: string indices must be integers
Hello, I'm trying to run metaflow on a docker alpine image with python and node. I've installed the metaflow required dependencies in the image along with other node requirements for my use case; and ran my script using --with batch:image=my-custom-image. It resulted in this error
2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] Setting up task environment.
2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] /usr/bin/python: No module named pip
2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] tar: can't open 'job.tar': No such file or directory
2020-12-23 00:14:08.205 [4747/start/30948 (pid 15530)] AWS Batch error:
2020-12-23 00:14:08.439 [4747/start/30948 (pid 15530)] Essential container in task exited This could be a transient error. Use @retry to retry.
2020-12-23 00:14:08.440 [4747/start/30948 (pid 15530)]
2020-12-23 00:14:08.791 [4747/start/30948 (pid 15530)] Task failed.My question is, pip and the required python dependencies are installed in the container so what is causing the No module named pip error? Thanks
14 replies
Anyone had problems with multi GPU on Aws Batch ? : I get like:
CannotStartContainerError: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr This could be a transient error. Use @retry to retry.1 GPU works fine
22 replies
CONDA_CHANNELS but that doesn't work for me.@tuulos You mentioned prepending
METAFLOW_RUN_ to the desired variable but this doesn't seem to bring the variable into the AWS Batch environment for me.@savingoyal You mentioned using the
environment decorator but I'm getting a linting error when attempting to use that. Then, when I use --no-pylint to override, none of my Flow steps work.I just want to be able to
os.environ.get a custom environment variable from within one of my Metaflow steps that was created in my local environment and passed to the AWS Batch environment. I feel like I'm missing something rather obvious.Thanks in advance for the help!
9 replies
hey guys – curious if anyone else would find value in exposing the batch job parameter for sharedMemorySize? It looks like the AWS batch team added the parameter towards the end of last year and it's a passthrough to docker run --shm-size, which can really speed up the performance of pytorch parallel dataloaders (especially to saturate multiple GPUs) and some boosting libraries.
ECS defaults the instance shm to 50% of memory allocation, but docker will only expose 64mb of that by default to running containers.
6 replies
2 replies
📣 Metaflow was just included in the Netflix's security bug bounty program! Find vulnerabilities in the code and get paid for it 💰(Or just enjoy Metaflow getting more secure over time)
https://bugcrowd.com/netflix/updates/59a4e5dc-5e79-4965-9289-ae5a0d9de044
Hey Metaflow! I have a pretty specific question:
I find myself having trouble running a flow on AWS Batch that uses a container with pre-installed Python libraries. I happen to be using conda to install a few extra libraries in this step but by doing so, it seems I now have a fragmented environment.
Any advice on how one can use a Docker container as a base environment and then seemingly add a few more packages in a specific step using conda?
The success criteria here would be to successfully import a package installed by the Docker image as well as a different package installed by the conda decorator
9 replies
--max-workers between the local runtime and when deployed via SFN, specifically when having nested foreach fanouts. Locally, the runtime will enforce the parallelization at the task level so it will never go beyond that, however the SFN concurrency limit is enforced per-split, so the nested fanout will result in an effective parallelism of max-workers^2. Similarly, normal fanouts in a SFN deployment are not rate limited. Not sure it’s worth explicitly stating this in the docs, but thought I’d mention it just in case
2 replies
I just noticed Batch has started hitting the Docker free tier rate limit. What’s the best way to mitigate this?
CannotPullContainerError: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limitAny chance we can get a copy of the Metaflow docker image hosted on the new Public ECR repos?
4 replies
6 replies
METAFLOW_INIT_. A parameter with a name like "my-param", which is otherwise perfectly valid for Metaflow when using the local runtime, will result in an error when running via SFN, because many shells won't allow env vars with dashes in the name.
3 replies
Hi, what is the way to run a nodejs process in the background in metaflow? I am running on batch using a custom docker image that has nodejs and python dependencies. The node app, once started, waits for a json post which is done by a task later in the metaflow python process.
The way to start the node app to is "npm run dev" however when I use os.system('npm run dev') , the metaflow process gets paused at "App listening on http: / / localhost :8888" (as below) since it starts the node app right away which is then waiting for the json on port 8888. However this will be calculated in a later metaflow step and posted via requests.post("http://localhost:8888/savings-report", json=self.json_structure)
2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] Starting report generator ...
2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > report-generator@1.0.0 dev /usr/src/app
2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > ts-node src/server.ts
2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] App listening on http://localhost:8888I would like to start the nodejs app using npm run dev via metaflow and leave it running in the background and continue to the next steps in metaflow
3 replies
Wondering if there's a more efficient way to implement the following design pattern directly in metaflow such that it would utilize multiprocessing to load and combine multiple dataframes after a foreach fanout:
df = [input.partition_df for input in inputs]
df = pd.concat(df, ignore_index=True, sort=False, copy=False)A hacky way that's coming to mind is to just use joblib.Parallel or metaflow's parallel_map to access the artifacts in parallel, but it feels a bit odd. This pattern may also be related to the roadmap effort to open source your all's in-house goodies for dataframes. I use partitioned parquet files in a couple places to split out data, pass references around, and load in parallel – but there's a couple use cases where I'd prefer to stay within the metaflow ecosystem if possible :smiley:
Curious what your all's thoughts are, and just want to make sure I'm not missing something like a clever usage of s3.get_many.
5 replies
I want to do this because I am thinking to create 2 different step functions from the same flow: MY_FLOW_STG and MY_FLOW_PRD and then to update these step functions through a pipeline when user pushes to specific branch
5 replies
12 replies
14 replies
--with batch it takes around 8 minutes) most of the time is consumed to bootstrap conda environment for each step before running it!!!is there something I missed here or is there any cache technique I should use to improve the flow performance on AWS?
6 replies
I was in our AWS Batch console and I noticed two jobs that were seeming stuck in RUNNING. The individual who kicked off those jobs says all his terminal sessions have been ended, even to go as far to restart his PC/sever internet connection.
I figure this is more of an AWS situation I'm debugging but has anyone witnessed flows being stuck in RUNNING?
I know the jobs will die when the timeout is reached, just want to understand what may have caused this
4 replies
