Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    ayorgo
    @ayorgo

    Hey Metaflow,
    How can I disable the timestamp in the printout? I tried to monkeypatch the logger as follows but it messed all the parameter taking up and didn't work

    from metaflow import cli
    from functools import partial
    cli.logger = partial(cli.logger, timestamp=False)

    Is there any other way?

    4 replies
    Christopher Wong
    @christopher-wong
    I’m testing out the new Metaflow scehduler for the first time (very excited!) but running into some issues with private conda packages. When I try and schedule, Metaflow obviously won’t be able to find my private conda package and fails to schedule. Previuosly, I manually installed the conda package on the EC2 I was using to run the job. Is there a workaround with Step functions?
    4 replies
    mkjacks5
    @mkjacks5

    from https://docs.metaflow.org/metaflow/tagging
    " if you have separate training and prediction flows in production, the prediction flow can access the previously built model as long as one exists in the same namespace"

    I have two such flows, but I can't figure out how to have them in the same namespace. I've tried --authorize but it seems it creates a unique production token (i.e. namespace("production:flow1-0-zjgv")) for every unique flow name. I'm able to get around this by changing namespaces inside the script, but it sounds from the documentation there should be a way to have them in the same namespace so that I can more easily access trained models from the traning flow when I run the prediction flow. Am I misunderstanding something here?

    1 reply
    Roman Kindruk
    @sappier

    Hi here, we're developing the plugin to run the Metaflow flows in the k8s using Argo Workflows. It's similar to the AWS Step Functions plugin but generates an Argo's WorkflowTemplate instead of the SFN's StateMachine. Also it adds an extra @argo decorator to specify k8s resources:

    @argo(image='tensorflow/tensorflow:2.2.1-gpu-py3', nodeSelector={'gpu': 'nvidia-tesla-v100'})
    @resources(gpu=1, cpu=2, memory=6000)
    @step
    def training(self):
        ...

    Would you be interested to make such plugin a part of the Metaflow project?

    7 replies
    Antoine Tremblay
    @hexa00
    I this this error familiar to anyone ? botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: 'SSLSocket' object has no attribute 'connection' , I get that with File "/metaflow/metaflow/datastore/s3.py", line 178, in save_metadata with requests-2.23.0 it needs 2.24 ? from the get_pinned_conda_libs it seems... will try
    6 replies
    Christopher Wong
    @christopher-wong

    I’ve been running into this issue quite a bit recently.

    AWS Batch Error: CannotCreateContainerError: Error response from daemon: devmapper: Thin Pool has 4115 free data blocks which is less than minimum required 4449 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior This could be a transient error. Use @retry to retry.

    Removing any EC2s controlled by Metaflow and letting the ASG create new ones seems to temporarily solve the problem but it keeps reappearing. Any advice on how to mitigate?

    5 replies
    Greg Hilston
    @GregHilston
    Hey guys, any thoughts as to what could be causing a conda Error: UnsatisfiableError when Bootstrapping a conda environment when running locally on OSX but not occur when running remotely on Batch?
    10 replies
    Ville Tuulos
    @tuulos
    many of you have asked how Netflix uses Metaflow internally. Here's finally a blog article that shares some details that we couldn't share earlier https://netflixtechblog.com/supporting-content-decision-makers-with-machine-learning-995b7b76006f
    1 reply
    Peter Wilton
    @ammodramus
    Hi folks, what is the suggested way of running 1000's of Batch jobs simultaneously? Attempting to run them using the standard python flow.py run causes the machine I'm running this on to go OOM after launching a Python process for each job. This is after setting --max-workers and --max-num-splits high enough to allow this many jobs.
    11 replies
    Antoine Tremblay
    @hexa00
    Is there a way to see the logs of step functions ? Looks like logs are not enabled by default... (For context my use case is running a job that takes 5 days... and that this unfortunately can't be ran from a laptop since the connection to the original caller may break.. So basically I'm trying to reproduce the normal batch workflow with step...) Let me know if there's a better way...
    5 replies
    Antoine Tremblay
    @hexa00

    I got this very weird issue running metaflow locally with a step that import pandas... while running the code out of metaflow it all works fine... but the same code in the same env with metaflow I get

     ImportError: cannot import name 'Collection' from 'typing' (/home/hexa/.cache/pypoetry/virtualenvs/nima-images-H6fn72k1-py3.7/lib/python3.7/site-packages/typing.py)

    Ideas ?

    5 replies
    Running in aws batch it runs fine too....
    Ville Väänänen
    @fortum-vaanavil
    Hi! I'd like to create a new custom Environment. Are there any examples on how to use the plugin mechanism?
    1 reply
    Antoine Tremblay
    @hexa00
    Hi there just wanted to say thanks to all of you working on metaflow, I was finally able to migrate key components of our arch to it ! and it now works great !
    Key lessons to date:
    • Using docker images is a lot less trouble then conda.. or to try to use pip via a custom decorator
    • Using patch_env is great to get debugging to work with session based aws keys
    • Dependency clashes due to metaflow actually being part of your app, can be really painful
      • Setting a minimum CPU to the cluster can leave with with a GPU instance running instead of pure CPU....
      • Using step functions for long running processes is still a bit painful, can't wait for improvements there :)
    5 replies
    Greg Hilston
    @GregHilston

    Hey Metaflow, one unexpected discovery I've stumbled upon is the orchestration of the DAG is performed locally, even when running on AWS Batch.

    Additionally, I created an access list for our AWS API Gateway that our Metaflow API uses, as one layer of security.

    This means that we have to leave our data scientists machines running and on the VPN, during long running training flows.

    Is there any configuration of Metaflow to allow the orchestration to be performed remotely and allow our development machines to be disconnected from the VPN or even shut down during execution?

    1 reply
    Christopher Wong
    @christopher-wong

    Has anyone run into intermmitent errors from the Metaflow service?

    failed (code 500): {“message”: “Internal server error”}

    This seems to happen on flows with multiple steps where the pipeline starts fine and runs a few steps, but then fails with the above error

    17 replies
    Malay Shah
    @malay95
    Hello all, we want to deploy our flows into production using metaflow and wanted to use fargate for the compute environment. Can we setup the job definition and other parameters required for submitting a fargate job to aws batch? Is there any documentation on how to setup fargate clusters for compute instead of ec2 on-demand compute environment?
    7 replies
    Malay Shah
    @malay95
    Hello all, I was wondering why we set ttl when we setup the dynamoDB table for step functions. And what would be a good value for that. I am not aware of the usage of DynamoDB for step functions.
    7 replies
    Malay Shah
    @malay95
    I am creating a fargate cluster for the metadata service, after following all the steps in the manual steps in the document. I see this error in the cloudwatch events:
    
    /migration_service/migration_server.py:17: DeprecationWarning: loop argument is deprecated
    
    app = web.Application(loop=loop)
    
    /migration_service/migration_server.py:28: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
    
    AttributeError: 'NoneType' object has no attribute 'cursor'
    
    /bin/sh: 1: metadata_service: not found
    4 replies
    Matej Války
    @enderstorm
    Hello all, I have just started working on data pipelines using excellent metaflow, but I am not sure how to make flows as steps, after Flow A, Flow B is run, in aws environment - batch + scheduled SFN. I tried to instantiate flows in one main flow, but that failed. Do you know any workarounds? Or have I to create single bigger flow, as every other flows are dependant on one the first flow?
    waz-mataz
    @waz-mataz
    Hi. I would like to build a custom docker image, with all the requirements for Metaflow and some other custom dependencies on top and use
    @batch(image='custom-image') . What would be the best way to build on top of the default image to ensure requirements for Metaflow are preserved? How does Metaflow install necessary dependencies on the default image?
    4 replies
    russellbrooks
    @russellbrooks
    kinda random but sharing in case it's useful for anyone else – when using SFN-based metaflow executions with fanout steps, the step before the fanout will fail if curl is not installed in the image. Seems to be used to lookup the dynamodb host. FWIW not an unreasonable dependency, and I was surprised to find that curlwasn't already baked into the continuumio/miniconda3:latest image. Once installing it in the docker image the dynamodb host resolution worked as expected.
    5 replies
    TDo13
    @TDo13

    Hello all, I'm trying to take a look at the artifacts associated with one of our SFN executions but when trying to call:

    Step(...).task.data

    We run into the following error:

    ServiceException: Metadata request (/flows/{Flow}/runs/{Run}/steps/start/tasks/{Task}/artifacts) failed (code 500): 500 Internal Server Error

    Looking at the logs from our metadata service, I see the following error:

    Traceback (most recent call last):
        File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
            resp = await task
        File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
            resp = await handler(request)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 140, in get_artifacts_by_task
            artifacts.body)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 355, in _filter_artifacts_by_attempt_id
            attempt_id = ArtificatsApi._get_latest_attempt_id(artifacts)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 349, in _get_latest_attempt_id
            if artifact['attempt_id'] > attempt_id:
    TypeError: string indices must be integers
    4 replies
    Itamar Turner-Trauring
    @itamarst
    @tuulos hi, stopping by from Hacker News
    32 replies
    waz-mataz
    @waz-mataz

    Hello, I'm trying to run metaflow on a docker alpine image with python and node. I've installed the metaflow required dependencies in the image along with other node requirements for my use case; and ran my script using --with batch:image=my-custom-image. It resulted in this error

    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] Setting up task environment.
    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] /usr/bin/python: No module named pip
    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
    2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
    2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] tar: can't open 'job.tar': No such file or directory
    2020-12-23 00:14:08.205 [4747/start/30948 (pid 15530)]     AWS Batch error:
    2020-12-23 00:14:08.439 [4747/start/30948 (pid 15530)]     Essential container in task exited This could be a transient error. Use @retry to retry.
    2020-12-23 00:14:08.440 [4747/start/30948 (pid 15530)]
    2020-12-23 00:14:08.791 [4747/start/30948 (pid 15530)] Task failed.

    My question is, pip and the required python dependencies are installed in the container so what is causing the No module named pip error? Thanks

    14 replies
    Antoine Tremblay
    @hexa00

    Anyone had problems with multi GPU on Aws Batch ? : I get like:

     CannotStartContainerError: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr This could be a transient error. Use @retry to retry.

    1 GPU works fine

    22 replies
    Sonu Patidar
    @skamdar
    @savingoyal Are you guys planning to support EMR on Metaflow as well?
    16 replies
    David Patschke
    @dpatschke
    Is there a way to pass a custom environment variable to a Metaflow AWS Batch job? I've seen several recommendations on this board and tried them all, but none of them work for me ... well none that don't expose the environment variable via the command-line. I think @russellbrooks showed an example with CONDA_CHANNELS but that doesn't work for me.
    @tuulos You mentioned prepending METAFLOW_RUN_ to the desired variable but this doesn't seem to bring the variable into the AWS Batch environment for me.
    @savingoyal You mentioned using the environment decorator but I'm getting a linting error when attempting to use that. Then, when I use --no-pylint to override, none of my Flow steps work.
    I just want to be able to os.environ.get a custom environment variable from within one of my Metaflow steps that was created in my local environment and passed to the AWS Batch environment. I feel like I'm missing something rather obvious.
    Thanks in advance for the help!
    11 replies
    russellbrooks
    @russellbrooks

    hey guys – curious if anyone else would find value in exposing the batch job parameter for sharedMemorySize? It looks like the AWS batch team added the parameter towards the end of last year and it's a passthrough to docker run --shm-size, which can really speed up the performance of pytorch parallel dataloaders (especially to saturate multiple GPUs) and some boosting libraries.

    ECS defaults the instance shm to 50% of memory allocation, but docker will only expose 64mb of that by default to running containers.

    6 replies
    seanv507
    @seanv507
    Hi is there an update on retroactive editing of tags? My use case is that we would want to label a run as "official" after human inspection. and to link flows together ( eg data preprocessing) followed by model run flow. I would like to tag the data_preprocessing flow used for a given model_run
    2 replies
    Vinicius Agostini
    @viagostini
    Hey guys, I was wondering if there is a way to make a Flow trigger another Flow, in order to reuse them as components of a bigger system or maybe if its on the roadmap, couldn't find anything about it
    12 replies
    NeeleshG
    @neeleshg
    Hi Guys,
    I want to try Metaflow IDS on AWS Infra.
    However when I checked AMI in Marketplace, it is updated in 2018.
    Do we have any updated AMI ?
    2 replies
    Ville Tuulos
    @tuulos

    📣 Metaflow was just included in the Netflix's security bug bounty program! Find vulnerabilities in the code and get paid for it 💰(Or just enjoy Metaflow getting more secure over time)

    https://bugcrowd.com/netflix/updates/59a4e5dc-5e79-4965-9289-ae5a0d9de044

    Greg Hilston
    @GregHilston

    Hey Metaflow! I have a pretty specific question:

    I find myself having trouble running a flow on AWS Batch that uses a container with pre-installed Python libraries. I happen to be using conda to install a few extra libraries in this step but by doing so, it seems I now have a fragmented environment.

    Any advice on how one can use a Docker container as a base environment and then seemingly add a few more packages in a specific step using conda?

    The success criteria here would be to successfully import a package installed by the Docker image as well as a different package installed by the conda decorator

    9 replies
    russellbrooks
    @russellbrooks
    Sharing a difference in the behavior of --max-workers between the local runtime and when deployed via SFN, specifically when having nested foreach fanouts. Locally, the runtime will enforce the parallelization at the task level so it will never go beyond that, however the SFN concurrency limit is enforced per-split, so the nested fanout will result in an effective parallelism of max-workers^2. Similarly, normal fanouts in a SFN deployment are not rate limited. Not sure it’s worth explicitly stating this in the docs, but thought I’d mention it just in case
    2 replies
    Christopher Wong
    @christopher-wong

    I just noticed Batch has started hitting the Docker free tier rate limit. What’s the best way to mitigate this?

    CannotPullContainerError: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

    Any chance we can get a copy of the Metaflow docker image hosted on the new Public ECR repos?

    4 replies
    Luis Arias
    @kaaloo
    Hello Metaflow community
    Just wanted to say I started working with Metaflow recently on processing some wikidump size datasets and after some battling with AWS Batch's Compute Environments and Launch Templates I managed to setup a working pipeline for us. The main challenge was understanding that the compute environment needed to be recreated each time I changed the launch template in spite of using the $Default version. Now the launch template takes care of using a much larger volume for the instances and allowing a lot more space for each docker container. Thanks so much for this wonderful piece of software! Now I'll be working on the next Flow in our pipeline....
    6 replies
    Ji Xu
    @xujiboy
    Hello, as I know that metaflow is supposed to be used in shell environment, I experimented and saw that I can also run metaflows in a notebook. May I know if there are some unforeseen, bad consequences for using it in a notebook?
    29 replies
    Matt Corley
    @corleyma
    I am having some issues with how the Metaflow Stepfunctions integration handles flow parameters. It seems like, only when executing a flow via SFN: flow parameters are converted (rather naively) into environment variables, by upcasing and then prepending with METAFLOW_INIT_. A parameter with a name like "my-param", which is otherwise perfectly valid for Metaflow when using the local runtime, will result in an error when running via SFN, because many shells won't allow env vars with dashes in the name.
    3 replies
    I was hoping to get some clarity on why this behavior exists in the first place for the StepFunctions integration, and then perhaps to strategize about the best approach to reconcile the allowable parameter names for stepfunctions and local runtime flows.
    Ayotomiwa Salau
    @AyonzOnTop
    Hello.
    Happy to be part of the Metaflow community.
    Cheers!
    2 replies
    Antoine Tremblay
    @hexa00
    Hi, is there a way to make a step-function flow in the user namespace ? It seeems like --namespace something has no effect ?
    The use case is that we're many users running different versions of the same flow with step functions....
    9 replies
    Antoine Tremblay
    @hexa00
    Is there a way to have kind of a fallback ECR repo ?
    Use case is that we have some custom images... but we'd still want to allow access to more general images like the default python one ....?
    3 replies
    waz-mataz
    @waz-mataz

    Hi, what is the way to run a nodejs process in the background in metaflow? I am running on batch using a custom docker image that has nodejs and python dependencies. The node app, once started, waits for a json post which is done by a task later in the metaflow python process.

    The way to start the node app to is "npm run dev" however when I use os.system('npm run dev') , the metaflow process gets paused at "App listening on http: / / localhost :8888" (as below) since it starts the node app right away which is then waiting for the json on port 8888. However this will be calculated in a later metaflow step and posted via requests.post("http://localhost:8888/savings-report", json=self.json_structure)

    2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] Starting report generator ...
    2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > report-generator@1.0.0 dev /usr/src/app
    2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > ts-node src/server.ts
    2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] App listening on http://localhost:8888

    I would like to start the nodejs app using npm run dev via metaflow and leave it running in the background and continue to the next steps in metaflow

    3 replies
    russellbrooks
    @russellbrooks

    Wondering if there's a more efficient way to implement the following design pattern directly in metaflow such that it would utilize multiprocessing to load and combine multiple dataframes after a foreach fanout:

    df = [input.partition_df for input in inputs]
    df = pd.concat(df, ignore_index=True, sort=False, copy=False)

    A hacky way that's coming to mind is to just use joblib.Parallel or metaflow's parallel_map to access the artifacts in parallel, but it feels a bit odd. This pattern may also be related to the roadmap effort to open source your all's in-house goodies for dataframes. I use partitioned parquet files in a couple places to split out data, pass references around, and load in parallel – but there's a couple use cases where I'd prefer to stay within the metaflow ecosystem if possible :smiley:

    Curious what your all's thoughts are, and just want to make sure I'm not missing something like a clever usage of s3.get_many.

    5 replies
    Savin
    @savingoyal
    :tada: Metaflow 2.2.6 (the newest release) is now available on pip and conda-forge. Changes include support for AWS Fargate as a compute backend for Metaflow on AWS, support for very wide workflows on AWS Step Functions and more.
    seanv507
    @seanv507
    Hi, is there anyway to specify the memory dynamically for batch jobs? eg if size of data =M in step X, allocate memory 5M in step X+1?
    4 replies
    Ahmad Houri
    @ahmad_hori_twitter
    Hi, is there a way to define step function name on aws to be different from the flow name when creating it?
    I want to do this because I am thinking to create 2 different step functions from the same flow: MY_FLOW_STG and MY_FLOW_PRD and then to update these step functions through a pipeline when user pushes to specific branch
    5 replies
    jrs2
    @jrs2
    Is there a way to specify a Docker image for a flow when running locally? I can see how to do it for Batch and have used that, but only see @conda for local dependency support.
    1 reply