Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Greg Hilston
    @GregHilston

    Hey Metaflow, one unexpected discovery I've stumbled upon is the orchestration of the DAG is performed locally, even when running on AWS Batch.

    Additionally, I created an access list for our AWS API Gateway that our Metaflow API uses, as one layer of security.

    This means that we have to leave our data scientists machines running and on the VPN, during long running training flows.

    Is there any configuration of Metaflow to allow the orchestration to be performed remotely and allow our development machines to be disconnected from the VPN or even shut down during execution?

    1 reply
    Christopher Wong
    @christopher-wong

    Has anyone run into intermmitent errors from the Metaflow service?

    failed (code 500): {“message”: “Internal server error”}

    This seems to happen on flows with multiple steps where the pipeline starts fine and runs a few steps, but then fails with the above error

    17 replies
    Malay Shah
    @malay95
    Hello all, we want to deploy our flows into production using metaflow and wanted to use fargate for the compute environment. Can we setup the job definition and other parameters required for submitting a fargate job to aws batch? Is there any documentation on how to setup fargate clusters for compute instead of ec2 on-demand compute environment?
    7 replies
    Malay Shah
    @malay95
    Hello all, I was wondering why we set ttl when we setup the dynamoDB table for step functions. And what would be a good value for that. I am not aware of the usage of DynamoDB for step functions.
    7 replies
    Malay Shah
    @malay95
    I am creating a fargate cluster for the metadata service, after following all the steps in the manual steps in the document. I see this error in the cloudwatch events:
    
    /migration_service/migration_server.py:17: DeprecationWarning: loop argument is deprecated
    
    app = web.Application(loop=loop)
    
    /migration_service/migration_server.py:28: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
    
    AttributeError: 'NoneType' object has no attribute 'cursor'
    
    /bin/sh: 1: metadata_service: not found
    4 replies
    Matej Války
    @enderstorm
    Hello all, I have just started working on data pipelines using excellent metaflow, but I am not sure how to make flows as steps, after Flow A, Flow B is run, in aws environment - batch + scheduled SFN. I tried to instantiate flows in one main flow, but that failed. Do you know any workarounds? Or have I to create single bigger flow, as every other flows are dependant on one the first flow?
    waz-mataz
    @waz-mataz
    Hi. I would like to build a custom docker image, with all the requirements for Metaflow and some other custom dependencies on top and use
    @batch(image='custom-image') . What would be the best way to build on top of the default image to ensure requirements for Metaflow are preserved? How does Metaflow install necessary dependencies on the default image?
    4 replies
    russellbrooks
    @russellbrooks
    kinda random but sharing in case it's useful for anyone else – when using SFN-based metaflow executions with fanout steps, the step before the fanout will fail if curl is not installed in the image. Seems to be used to lookup the dynamodb host. FWIW not an unreasonable dependency, and I was surprised to find that curlwasn't already baked into the continuumio/miniconda3:latest image. Once installing it in the docker image the dynamodb host resolution worked as expected.
    5 replies
    TDo13
    @TDo13

    Hello all, I'm trying to take a look at the artifacts associated with one of our SFN executions but when trying to call:

    Step(...).task.data

    We run into the following error:

    ServiceException: Metadata request (/flows/{Flow}/runs/{Run}/steps/start/tasks/{Task}/artifacts) failed (code 500): 500 Internal Server Error

    Looking at the logs from our metadata service, I see the following error:

    Traceback (most recent call last):
        File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
            resp = await task
        File "/opt/latest/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
            resp = await handler(request)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 140, in get_artifacts_by_task
            artifacts.body)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 355, in _filter_artifacts_by_attempt_id
            attempt_id = ArtificatsApi._get_latest_attempt_id(artifacts)
        File "/opt/latest/lib/python3.7/site-packages/services/metadata_service/api/artifact.py", line 349, in _get_latest_attempt_id
            if artifact['attempt_id'] > attempt_id:
    TypeError: string indices must be integers
    4 replies
    Itamar Turner-Trauring
    @itamarst
    @tuulos hi, stopping by from Hacker News
    32 replies
    waz-mataz
    @waz-mataz

    Hello, I'm trying to run metaflow on a docker alpine image with python and node. I've installed the metaflow required dependencies in the image along with other node requirements for my use case; and ran my script using --with batch:image=my-custom-image. It resulted in this error

    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] Setting up task environment.
    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] /usr/bin/python: No module named pip
    2020-12-23 00:14:06.109 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
    2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] sh: 5: unknown operand
    2020-12-23 00:14:08.204 [4747/start/30948 (pid 15530)] [a3c2939b-9256-41e5-8c7e-f076261e2739] tar: can't open 'job.tar': No such file or directory
    2020-12-23 00:14:08.205 [4747/start/30948 (pid 15530)]     AWS Batch error:
    2020-12-23 00:14:08.439 [4747/start/30948 (pid 15530)]     Essential container in task exited This could be a transient error. Use @retry to retry.
    2020-12-23 00:14:08.440 [4747/start/30948 (pid 15530)]
    2020-12-23 00:14:08.791 [4747/start/30948 (pid 15530)] Task failed.

    My question is, pip and the required python dependencies are installed in the container so what is causing the No module named pip error? Thanks

    14 replies
    Antoine Tremblay
    @hexa00

    Anyone had problems with multi GPU on Aws Batch ? : I get like:

     CannotStartContainerError: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr This could be a transient error. Use @retry to retry.

    1 GPU works fine

    22 replies
    Sonu Patidar
    @skamdar
    @savingoyal Are you guys planning to support EMR on Metaflow as well?
    16 replies
    David Patschke
    @dpatschke
    Is there a way to pass a custom environment variable to a Metaflow AWS Batch job? I've seen several recommendations on this board and tried them all, but none of them work for me ... well none that don't expose the environment variable via the command-line. I think @russellbrooks showed an example with CONDA_CHANNELS but that doesn't work for me.
    @tuulos You mentioned prepending METAFLOW_RUN_ to the desired variable but this doesn't seem to bring the variable into the AWS Batch environment for me.
    @savingoyal You mentioned using the environment decorator but I'm getting a linting error when attempting to use that. Then, when I use --no-pylint to override, none of my Flow steps work.
    I just want to be able to os.environ.get a custom environment variable from within one of my Metaflow steps that was created in my local environment and passed to the AWS Batch environment. I feel like I'm missing something rather obvious.
    Thanks in advance for the help!
    9 replies
    russellbrooks
    @russellbrooks

    hey guys – curious if anyone else would find value in exposing the batch job parameter for sharedMemorySize? It looks like the AWS batch team added the parameter towards the end of last year and it's a passthrough to docker run --shm-size, which can really speed up the performance of pytorch parallel dataloaders (especially to saturate multiple GPUs) and some boosting libraries.

    ECS defaults the instance shm to 50% of memory allocation, but docker will only expose 64mb of that by default to running containers.

    6 replies
    seanv507
    @seanv507
    Hi is there an update on retroactive editing of tags? My use case is that we would want to label a run as "official" after human inspection. and to link flows together ( eg data preprocessing) followed by model run flow. I would like to tag the data_preprocessing flow used for a given model_run
    2 replies
    Vinicius Agostini
    @viagostini
    Hey guys, I was wondering if there is a way to make a Flow trigger another Flow, in order to reuse them as components of a bigger system or maybe if its on the roadmap, couldn't find anything about it
    12 replies
    NeeleshG
    @neeleshg
    Hi Guys,
    I want to try Metaflow IDS on AWS Infra.
    However when I checked AMI in Marketplace, it is updated in 2018.
    Do we have any updated AMI ?
    2 replies
    Ville Tuulos
    @tuulos

    📣 Metaflow was just included in the Netflix's security bug bounty program! Find vulnerabilities in the code and get paid for it 💰(Or just enjoy Metaflow getting more secure over time)

    https://bugcrowd.com/netflix/updates/59a4e5dc-5e79-4965-9289-ae5a0d9de044

    Greg Hilston
    @GregHilston

    Hey Metaflow! I have a pretty specific question:

    I find myself having trouble running a flow on AWS Batch that uses a container with pre-installed Python libraries. I happen to be using conda to install a few extra libraries in this step but by doing so, it seems I now have a fragmented environment.

    Any advice on how one can use a Docker container as a base environment and then seemingly add a few more packages in a specific step using conda?

    The success criteria here would be to successfully import a package installed by the Docker image as well as a different package installed by the conda decorator

    9 replies
    russellbrooks
    @russellbrooks
    Sharing a difference in the behavior of --max-workers between the local runtime and when deployed via SFN, specifically when having nested foreach fanouts. Locally, the runtime will enforce the parallelization at the task level so it will never go beyond that, however the SFN concurrency limit is enforced per-split, so the nested fanout will result in an effective parallelism of max-workers^2. Similarly, normal fanouts in a SFN deployment are not rate limited. Not sure it’s worth explicitly stating this in the docs, but thought I’d mention it just in case
    2 replies
    Christopher Wong
    @christopher-wong

    I just noticed Batch has started hitting the Docker free tier rate limit. What’s the best way to mitigate this?

    CannotPullContainerError: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

    Any chance we can get a copy of the Metaflow docker image hosted on the new Public ECR repos?

    4 replies
    Luis Arias
    @kaaloo
    Hello Metaflow community
    Just wanted to say I started working with Metaflow recently on processing some wikidump size datasets and after some battling with AWS Batch's Compute Environments and Launch Templates I managed to setup a working pipeline for us. The main challenge was understanding that the compute environment needed to be recreated each time I changed the launch template in spite of using the $Default version. Now the launch template takes care of using a much larger volume for the instances and allowing a lot more space for each docker container. Thanks so much for this wonderful piece of software! Now I'll be working on the next Flow in our pipeline....
    6 replies
    Ji Xu
    @xujiboy
    Hello, as I know that metaflow is supposed to be used in shell environment, I experimented and saw that I can also run metaflows in a notebook. May I know if there are some unforeseen, bad consequences for using it in a notebook?
    29 replies
    Matt Corley
    @corleyma
    I am having some issues with how the Metaflow Stepfunctions integration handles flow parameters. It seems like, only when executing a flow via SFN: flow parameters are converted (rather naively) into environment variables, by upcasing and then prepending with METAFLOW_INIT_. A parameter with a name like "my-param", which is otherwise perfectly valid for Metaflow when using the local runtime, will result in an error when running via SFN, because many shells won't allow env vars with dashes in the name.
    3 replies
    I was hoping to get some clarity on why this behavior exists in the first place for the StepFunctions integration, and then perhaps to strategize about the best approach to reconcile the allowable parameter names for stepfunctions and local runtime flows.
    Ayotomiwa Salau
    @AyonzOnTop
    Hello.
    Happy to be part of the Metaflow community.
    Cheers!
    2 replies
    Antoine Tremblay
    @hexa00
    Hi, is there a way to make a step-function flow in the user namespace ? It seeems like --namespace something has no effect ?
    The use case is that we're many users running different versions of the same flow with step functions....
    9 replies
    Antoine Tremblay
    @hexa00
    Is there a way to have kind of a fallback ECR repo ?
    Use case is that we have some custom images... but we'd still want to allow access to more general images like the default python one ....?
    3 replies
    waz-mataz
    @waz-mataz

    Hi, what is the way to run a nodejs process in the background in metaflow? I am running on batch using a custom docker image that has nodejs and python dependencies. The node app, once started, waits for a json post which is done by a task later in the metaflow python process.

    The way to start the node app to is "npm run dev" however when I use os.system('npm run dev') , the metaflow process gets paused at "App listening on http: / / localhost :8888" (as below) since it starts the node app right away which is then waiting for the json on port 8888. However this will be calculated in a later metaflow step and posted via requests.post("http://localhost:8888/savings-report", json=self.json_structure)

    2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] Starting report generator ...
    2021-01-26 00:29:49.835 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > report-generator@1.0.0 dev /usr/src/app
    2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] > ts-node src/server.ts
    2021-01-26 00:29:49.836 [4816/start/31201 (pid 78223)] [94fd75e2-9b6a-4c13-87a9-57f6e6d4b811] App listening on http://localhost:8888

    I would like to start the nodejs app using npm run dev via metaflow and leave it running in the background and continue to the next steps in metaflow

    3 replies
    russellbrooks
    @russellbrooks

    Wondering if there's a more efficient way to implement the following design pattern directly in metaflow such that it would utilize multiprocessing to load and combine multiple dataframes after a foreach fanout:

    df = [input.partition_df for input in inputs]
    df = pd.concat(df, ignore_index=True, sort=False, copy=False)

    A hacky way that's coming to mind is to just use joblib.Parallel or metaflow's parallel_map to access the artifacts in parallel, but it feels a bit odd. This pattern may also be related to the roadmap effort to open source your all's in-house goodies for dataframes. I use partitioned parquet files in a couple places to split out data, pass references around, and load in parallel – but there's a couple use cases where I'd prefer to stay within the metaflow ecosystem if possible :smiley:

    Curious what your all's thoughts are, and just want to make sure I'm not missing something like a clever usage of s3.get_many.

    5 replies
    Savin
    @savingoyal
    :tada: Metaflow 2.2.6 (the newest release) is now available on pip and conda-forge. Changes include support for AWS Fargate as a compute backend for Metaflow on AWS, support for very wide workflows on AWS Step Functions and more.
    seanv507
    @seanv507
    Hi, is there anyway to specify the memory dynamically for batch jobs? eg if size of data =M in step X, allocate memory 5M in step X+1?
    4 replies
    Ahmad Houri
    @ahmad_hori_twitter
    Hi, is there a way to define step function name on aws to be different from the flow name when creating it?
    I want to do this because I am thinking to create 2 different step functions from the same flow: MY_FLOW_STG and MY_FLOW_PRD and then to update these step functions through a pipeline when user pushes to specific branch
    5 replies
    jrs2
    @jrs2
    Is there a way to specify a Docker image for a flow when running locally? I can see how to do it for Batch and have used that, but only see @conda for local dependency support.
    1 reply
    Mehmet Catalbas
    @baratrion
    What is the best way to profile memory usage step by step within a flow (better yet line-by-line within a step?) memory_profiler @profile decorator does not work well with foreach steps, so is the best approach to copy steps to isolated functions to different scripts and run memory profiling on them?
    12 replies
    seanv507
    @seanv507
    @baratrion you might want to look at https://pythonspeed.com/products/filmemoryprofiler/, the author @itamarst has reached out here ... its focussing on peak memory
    14 replies
    Ahmad Houri
    @ahmad_hori_twitter
    Hi, I have a question regarding the performance of running the batch jobs in AWS, I run a simple flow (HelloWorld) on my machine which contains 3 steps, the execution on my machine took around 23 seconds while on AWS (when I run the same flow --with batch it takes around 8 minutes) most of the time is consumed to bootstrap conda environment for each step before running it!!!
    is there something I missed here or is there any cache technique I should use to improve the flow performance on AWS?
    6 replies
    Greg Hilston
    @GregHilston

    I was in our AWS Batch console and I noticed two jobs that were seeming stuck in RUNNING. The individual who kicked off those jobs says all his terminal sessions have been ended, even to go as far to restart his PC/sever internet connection.

    I figure this is more of an AWS situation I'm debugging but has anyone witnessed flows being stuck in RUNNING?

    I know the jobs will die when the timeout is reached, just want to understand what may have caused this

    4 replies
    joe153
    @joe153
    Hi, I am trying to include a number of json/sql files in the conda package. I have MANIFEST.in file specifying the files and also setup.py has include_package_data=True but I am not seeing them. What am I missing? How do I include them in the conda package?
    6 replies
    Nimar Arora
    @nimar
    Hi, I am just trying to follow along with the tutorial and in 08-autopilot my AWS Batch job fails with "ModuleNoteFoundError: No module named 'pandas'" on stats.py, line 41. Looking at the code I'm not sure how this tutorial is expected to work since there is no @conda decorator to install the pandas library in 02-statistics/stats.py.
    3 replies
    Nimar Arora
    @nimar

    The tutorial 4 seems to be failing attempting to create a conda environment. The funny thing is that if I run that command directly it seems to succeed. Not sure how to get the conda errors:

    python 04-playlist-plus/playlist.py --environment=conda runMetaflow 2.2.6 executing PlayListFlow for user:...
    Validating your flow...
        The graph looks good!
    Running pylint...
        Pylint is happy!
    Bootstrapping conda environment...(this could take a few minutes)
        Conda ran into an error while setting up environment.:
        Step: start, Error: command '['/opt/miniconda/condabin/conda', 'create', '--yes', '--no-default-packages', '--name', 'metaflow_PlayListFlow_linux-64_c08336d0946efed6e92f165475dfc0d181f64361', '--quiet', b'python==3.8.5', b'click==7.1.2', b'requests==2.24.0', b'boto3==1.17.0', b'coverage==5.4', b'pandas==0.24.2']' returned error (-9): b''

    Note that the following command succeeds:

    /opt/miniconda/condabin/conda create --yes --no-default-packages --name metaflow_PlayListFlow_linux-64_c08336d0946efed6e92f165475dfc0d181f64361 --quiet python==3.8.5 click==7.1.2 requests==2.24.0 boto3==1.17.0 coverage==5.4 pandas==0.24.2

    Note: that I had to make a few minor changes to the demo to refer to Python 3.8.5 and to add a dependency to more recent versions of boto3 and coverage than what metaflow was requesting otherwise the generated conda create command would fail even on the command line.

    1 reply
    joe153
    @joe153
    Hi, I am having a problem using Metaflow 2.2.6 and Fargate as the compute environment when foreach is used. What works fine with EC2 doesn't work with Fargate. Here is the error message: ... File "/metaflow/metaflow/plugins/aws/step_functions/step_functions_decorator.py", line 54, in task_finished self._save_foreach_cardinality(os.environ['AWS_BATCH_JOB_ID'], ... requests.exceptions.ConnectionError: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/placement/availability-zone/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7c0f12e3d0>: Failed to establish a new connection: [Errno 22] Invalid argument'))
    7 replies
    Ritesh Agrawal
    @ragrawal
    hi, I am trying to leverage metaflow to train and deploy models on sagemaker. I am able to train the model but not able to find relevant documentation on how to deploy models. Ideally I would like to create a docker container with proper environment and all the supporting files and then deploy the docker container either on sagemaker or on our kubernetes cluster. What I am missing is once the pipeline is execute successfully, how can I get the environment, supporting files, model files
    13 replies
    Anirudh Kaushik
    @anirudh-k
    Hi! What's the best way to handle a potentially empty list for a foreach step?
    7 replies
    Ritesh Agrawal
    @ragrawal
    where to specify import statements. Assuming I have a train step that requires sklearn and I am using @conda to install the package. Should I define import sklearn statement inside the step or it can be outside the class definition
    6 replies
    Ritesh Agrawal
    @ragrawal
    I am getting access denied to following s3 folder: "s3://.../metaflow/conda" as it doesn't exists. Is there anything I need to do in order to create this S3 Key ?
    4 replies
    Ritesh Agrawal
    @ragrawal
    why code has all the metaflow examples in it
    4 replies
    Kyle Smith
    @smith-kyle
    If a step both creates a bunch of tasks with a foreach and branches to another step, will all the tasks created by this step execute in parallel?
    9 replies