Metaflow/community

sebrahimi1988

@sebrahimi1988

Hi guys, when using @conda can we install our own packages from a local repository?

4 replies

Bahattin Çiniç

@bahattincinic

Hi Guys,

I'm trying to add an integration/unit test for MetaFlow Flows. (I read https://gitter.im/metaflow_org/community?at=5e5e87fcca2f5a558d62ad96 https://docs.metaflow.org/internals-of-metaflow/testing-philosophy these docs.)

When I tried to use flow with use_cli=False mode, I got a could not get source code error. Example code and error are below;

import unittest
from flows import OptimizationFlow 

class TestFLow(unittest.TestCase):
    def test_flow(self):
        flow = OptimizationFlow(use_cli=False)

Error:

Traceback (most recent call last):
  File "/Users/bahattincinic/Projects/xx/xx/xx/tests/cases/flow.py", line 16, in test_flow
    flow = OptimizationFlow(use_cli=False)
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/site-packages/metaflow/flowspec.py", line 70, in __init__
    self._graph = FlowGraph(self.__class__)
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/site-packages/metaflow/graph.py", line 132, in __init__
    self.nodes = self._create_nodes(flow)
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/site-packages/metaflow/graph.py", line 139, in _create_nodes
    tree = ast.parse(inspect.getsource(module)).body
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/inspect.py", line 973, in getsource
    lines, lnum = getsourcelines(object)
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/inspect.py", line 955, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/local/anaconda3/envs/xx/lib/python3.7/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

I had a chance to read some internal code after the error. I realized that flow cannot run outside of its own file. Because MetaFlow trying to detect the flow graph with AST.

I saw that MetaFlow uses subprecess.call (https://github.com/Netflix/metaflow/blob/master/test/core/run_tests.py#L106) and dynamic code generation (https://github.com/Netflix/metaflow/blob/master/test/core/metaflow_test/formatter.py#L80).

is this best practice?

Should I use like subprocess.call("python flow.py run ...params...")

Thanks.

7 replies

joe153

@joe153

Question on publishing scheduled step functions. When creating a step function with "step-functions create", how can I provide my defined parameters? I understand you can manually provide the parameters in AWS console like {"Parameters" : "{\"alpha\": 0.5}"} but not clear how to do it for the CloudWatch scheduled rule.

14 replies

joe153

@joe153

Question on the default AWS region. When a batch job runs from the step function, how does it know which AWS region to download the code package from? How can I provide my preferred region? Below is the error I am getting. I can fix this by updating the deployed step function and providing the AWS_DEFAULT_REGION environment variable manually but that's not good.

Setting up task environment.
Downloading code package.
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

17 replies

nspies-celsiustx

@nspies-celsiustx

Is there any way to access the tags from within a step? I see the user and namespace but can't find additional tags.

4 replies

Greg Hilston

@GregHilston

Hey guys, I have two questions regarding Metaflow's best practices:

Do you guys recommend using Metaflow for large batch transform jobs? I'm wondering if there's some threshold of data size where the recommended approach may be to use AWS SageMaker for batch transform instead of Metaflow.
Is there a recommended approach for logging steps that are ran remote on AWS Batch?

Thanks :)

10 replies

Christopher Wong

@christopher-wong

I’m running into an issue where after a Batch task starts, I’m getting the error

Batch job error:
TypeError(“‘NoneType’ object is not iterable”)
…

Task failed.

23 replies

Christopher Wong

@christopher-wong

3 replies

Sonu Patidar

@skamdar

@savingoyal can you please tell me something about this error: File "/home/ec2-user/.local/lib/python3.6/site-packages/metaflow/datastore/datastore.py", line 429, in <lambda>
2020-09-11 05:22:34.014 [8/start/16 (pid 2995)] transformable_obj.transform(lambda x: pickle.dumps(x, protocol=2))
2020-09-11 05:22:34.014 [8/start/16 (pid 2995)] TypeError: can't pickle _thread.RLock objects

9 replies

Sonu Patidar

@skamdar

I am using docker container on aws batch and it is not able to find job.tar file. My docker file ends with WORKDIR /srv/ can you tell me where metaflow puts job.tar file?

29 replies

mishra-avinash

@mishra-avinash

hello team,
i am running a simple Flow with conda decorater @conda(libraries={"beautifulsoup4": "4.9.1", "s3fs": "0.5.1", "pandas": "1.1.2"})
i am running into error Error: UnsatisfiableError: The following specifications were found to be incompatible with each other:
i checked a previous post in this channel, with the same error and later there was one PR to solve this issue and fix was included in latest release.
even though i am getting this error, could someone help me? thanks

15 replies

joe153

@joe153

What's the recommended approach to use sensitive information such as passwords? For example, if I want to connect to a database to get some data out but I don't want the password to be in the code. I understand the custom environment variable is not supported.

5 replies

Philippe Ombredanne

@pombredanne

Howdy... as a FYI we just made a first release of this new tool https://github.com/nexB/scancode.io that makes extensive use of metaflow for code analysis pipelines for origin and licenses such as these https://github.com/nexB/scancode.io/tree/main/scanpipe/pipelines

I want to thank the metaflow team for making this possible :bow:

@savingoyal and @romain-intel in partcular ... :bow:

Note that's a tad off broadway from your standard use case but still works nicely for us

6 replies

Valay Dave

@valayDave

Hey,

I was recently trying to store the data in the flow MetaflowData object which gets stored at the end of a flow. I was using setattr(self,some_id,dict) to store a large number of objects as a part of the MetaflowData object at the end of the flow. When loading the values from the datum via getattr the first 1000 elements finish within a 1 second. But after that, it took 4 minutes to load the next 3000. I am just simply iterating through the values via getattr . I am assuming that it's getting the data via the metadata and picklising objects. But what could be the reason for such latency gaps ?

4 replies

roverhuang

@styxhuang

Hey guys, I am a newbie using metaflow, just finished first three tutorials. I have following confusions when I am using metaflow. (1) In metaflow, document says using Includefile to read a .csv local file. I noticed, "IncludeFile" will generate a compress file in the .metaflow folder, but the speed of using pd.read_csv to read this compress file is slower than using pd.read_csv to directly read original csv file especially when the .csv file's size is large. Then what's the reason that metaflow using this "IncludeFile" function to pre-read the file? (2) If I want to read a like 7Gb csv file, directly using pd.read_csv is fine. However, if I use pd.read_csv under the metaflow, after couple of minutes, it will give me like "memory outflow" error. What's the reason of this error, how to avoid this and if there is any way to read this kind of files faster? Thanks!

roverhuang

@styxhuang

Hey guys, I am a newbie using metaflow, just finished first three tutorials. I have following confusions when I am using metaflow. (1) In metaflow, document says using Includefile to read a .csv local file. I noticed, "IncludeFile" will generate a compress file in the .metaflow folder, but the speed of using pd.read_csv to read this compress file is slower than using pd.read_csv to directly read original csv file especially when the .csv file's size is large. Then what's the reason that metaflow using this "IncludeFile" function to pre-read the file? (2) If I want to read a like 7Gb csv file, directly using pd.read_csv is fine. However, if I use pd.read_csv under the metaflow, after couple of minutes, it will give me like "memory outflow" error. What's the reason of this error, how to avoid this and if there is any way to read this kind of files faster? Thanks!

Plus I used the resouces decorator (memory=16000, cpu=8)to ask more resouces, but it still didn't work

7 replies

Philippe Ombredanne

@pombredanne

MO is that your large CSV file may end up being pickled?
Try to post snippets of your pipeline and the error messages/stack trace in a Gist

Greg Hilston

@GregHilston

Hey everyone, what's Metaflow's recommended approach for installing a dependency in a remote Conda environment when the package does not exist in Conda or the specific version does not exist in Conda, but does exist in Pip?

From the documentation I've found that I have two options:

Perform a os.system('pip install my_package') in my Step's code which looks like it should work but does not look like a great solution.
Download the code into my source code directory and import the package from the file system. Also seems like it'd work but not a great solution.

Are there any options I'm not considering? Perhaps a cleaner approach?

Philippe Ombredanne

@pombredanne

@GregHilston see Netflix/metaflow#24

2 replies

Gustav Svensk

@GSvensk

Hi, I'm not able to view the logs from the metadata_service when running it in a docker container. A minimal example can be achieved by creating a virtual env, entering it and running pip install, I also did

docker pull netflixoss/metaflow_metadata_service
ds up -d

ds logs
gives me
metadata_service | 2020/09/21 15:33:31 OK 1_create_tables.sql metadata_service | 2020/09/21 15:33:31 OK 20200603104139_add_str_id_cols.sql metadata_service | 2020/09/21 15:33:31 goose: no migrations to run. current version: 20200603104139 my_postgres | The files belonging to this database system will be owned by user "postgres". my_postgres | This user must also own the server process. ...
But not serving on ('0.0.0.0', 8080)which I would expect.

On MacOs catalina 10.15.6

2 replies

roverhuang

@styxhuang

Hey guys, I noticed metaflow did a great job on communicating between local machine and AWS. If I want to implement other cloud system, like Azure Blob Storage, how can I also implement some decorators (like batch). I know s3 module is used in the batch decorator , is there a way to make a change on that? Thanks!

2 replies

Greg Hilston

@GregHilston

Hey guys, is there a recommended way to pass secrets to a remote Batch job? I found this being asked in the Gitter back in 2019-12-07 here:

https://gitter.im/metaflow_org/community?at=5dee3c74d64a052fb6e2afaf

Basically the same question but with a very specific focus on the values of the environment being treated as sensitive. Therefore the environment_decorator or any other solution that stores the secret in code does not work.

I read that Savin said the AWS folks are able to seamlessly switch between running locally and remote so there must be a solution to this. Just wanted to ask you guys before trying to come up with any custom way.

Thanks!

2 replies

Peter Wilton

@ammodramus

Hi all, I'm working on converting a legacy pipeline to Metaflow and was wondering whether there is any way to do something like the following.

@step
def map_step(self):
    self.vars = ['a', 'b']
    self.next(self.do_compute, foreach='vars')

@step
def do_compute(self):
    self.var = self.input
    self.artifact1 = do_something(self.var)
    self.artifact2 = do_something_else(self.var)
    self.artifact3 = do_something_else_yet(self.var)
    self.next(self.join_step)

@step
def join_step(self, inputs):
    self.artifact_dict = dict()
    for inp in inputs:
        self.artifact_dict[inp.var] = inp

I was hoping that this would give me programmatic, lazily-loading access to the artifacts computed in do_compute for each value of var (a la self.artifact_dict['a'].artifact1), but of course I am getting this error message:

Flows can't be serialized. Maybe you tried to assign self or one of the inputs to an attribute? Instead of serializing the whole flow, you should choose specific attributes, e.g. input.some_var, to be stored.

Is there a recommended way to achieve this programmatic, lazy access? I see a workaround programmatically defining names and calling setattr and getattr, and searching through this gitter's history, this approach seems to have been recommended before. Is that still the recommended approach? Thanks!

21 replies

Andrew Achkar

@hajapy

Hello, I’m trying to determine what is the best option for having ci testing of my metaflow workflows. Are there examples I could follow on how to use pytest or the unittest framework to execute the flow (locally) and ensure outputs/artifacts are as expected?

9 replies

ayorgo

@ayorgo

Hey Metaflow,
I'm having an issue with running one of my flows on AWS Batch. The issue is as follows

mkdir: cannot create directory ‘metaflow’: Permission denied
/bin/sh: 1: [: -le: unexpected operator
tar: job.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

I run it on a pre-built image hosted on ECR. The Dockerfile contains a WORKDIR command which points at /home/my_proj/code. I can successfully build my image locally, bash into it and mkdir metaflow (under the default /home/my_proj/code/ directory) without an issue.
What I suspect might be happening is that the WORKDIR statement is somehow ignored and the Metaflow command ["/bin/sh","-c","set -e ... is run from within /.
It's worth noting that I have several flows running on AWS Batch already with no problem at all. Their Dockerfiles are almost identical to the one that is having the problem.

Not really sure if it's a Metaflow issue but hoping for somebody to have seen this already.
Thank you.

7 replies

ayorgo

@ayorgo

Hello again, Metaflow,
Is there a way to run flows on AWS Batch in a detached mode so I can start my run, close the lid of my laptop and go home without interrupting the execution to check its status later from the AWS Management Console? Or does it need access to my local disk to be able to package up the next step for execution?

3 replies

Youliang Yu

@elv-youliangyu

Hey Metaflow,
I am new to Metaflow, and trying to update a parameter in a step, but get "AttributeError: can't set attribute".
Here is the snippet:

class TestFlow(FlowSpec):
    param = Parameter(
        'param',
        type=str,
        help='test parameter',
        default='OD'
    )
    @step
    def start(self):
        self.param = self.param.lower()
        ...
    ....

This seems a common use cases. Probably there's some mistake in my usage. What am I missing?

4 replies

David Patschke

@dpatschke

@savingoyal @tuulos I want to throw something a little wild out here. I see in a couple of messages that you mention a lot of development of Flows at Netflix happen within notebooks. Have you all ever explored creating a jupyterlab-extension for Metaflow in the same vein as something like Kubeflow Kale?

7 replies

Denis Maciel

@denismaciel

Hi there,

I have a Flow that is supposed to run completely on batch. I'd like to run one of the steps in a different Docker container than the default one. Is it possible to run something like this:

# pipeline.py
from metaflow import FlowSpec, step, batch

class TestFlow(FlowSpec):

    @batch(image='python:3.6.12-buster')
    @step
    def start(self):
        import sys
        print(sys.version)
        self.next(self.end)

    @step
    def end(self):
        import sys
        print(sys.version)

if __name__ == "__main__":
    TestFlow()

with the following command: python pipeline.py run --with batch

Here the end step should run with the default image and the start step with python:3.6.12-buster.

3 replies

Andrew Achkar

@hajapy

Hi, I have a use case I’m not sure how to handle best with metaflow. I have a flow that is largely identical but that the team iterates one of the steps quite frequently. This can either be in the form of a new conda package or a new docker container. What would be the recommended way to allow this step to take different versions of the package/image for that step, without having to modify the flow code (so perhaps by providing a cli arg or env var when triggering the flow)?

6 replies

David Patschke

@dpatschke

@tuulos Would you be able to share what is happening from a memory perspective when artifacts are being preserved at the end of a task? I have a pretty large local machine (128GB) and I'm running multiple workers in parallel using 'foreach'. I'm seeing what looks to be an effective rapid doubling in memory right before the parallel tasks complete ... nearly depleting the available memory on my machine. Actually, I've had to significantly throttle down 'max-workers' due to this phenomenon (down to 4). Then, when the task(s) complete, the memory quickly goes back down. In contrast, if I create the same 4 artifacts within a jupyter notebook I see somewhere between 40-50 GB of memory being consumed. I looked at the Github issues but couldn't find any that exactly discussed what I'm seeing. Any suggestions?

5 replies

Greg Hilston

@GregHilston

Hey Metaflow, I stumbled upon a situation that I was hoping you guys could comment on. I'm trying out Metaflow in a code base that already exists, which provides numerous utility files and functionality.

I was read ingthrough this page on managing external libraries (https://docs.metaflow.org/metaflow/dependencies) and was wondering if there is any other way to let a Metaflow flow import these utility files without having to copy them from all over the repository into the same folder the flow is defined in.

I'm aware of symbolic links and file system approaches but was wondering if there was any other Metaflow approach for a scenario like this

3 replies

Ji Xu

@xujiboy

Hi, I recently upgraded metaflow from 2.0.1 to 2.2.3, and when I execute a parameterized flow I got the following error which I haven't seen before (truncated to the last few lines):

...
File "/home/ji.xu/.conda/envs/logan_env/lib/python3.7/site-packages/metaflow/includefile.py", line 229, in convert
    param_ctx = context_proto._replace(parameter_name=self.parameter_name)
AttributeError: 'NoneType' object has no attribute '_replace'

Any suggestions?

7 replies

Andrew Achkar

@hajapy

Hi, I've got another question from some metaflow trials. Say I have a conda step and want to run in batch, but my local machine is a mac and the batch environment is linux. It is conceivable that the batch execution conda environment is not buildable on mac (eg. specifying nvidia::cudatoolkit as a conda library). Why does this step attempt to create the conda environment locally before sending the step to batch and hence fail the flow? Is there any way to bypass this?

6 replies

Timothy Do

@TDo13

Hi again! I've been running into a particularly interesting issue when trying to use a docker image from our private repository. It looks like metaflow mistakenly identifies a registry for image paths like: foo/bar/baz:latest

5 replies

russellbrooks

@russellbrooks

Question for you all – is there a way to specify to use "any production namespace" within the client API? For example, when accessing the results of one flow in a subsequent flow, the namespaces work very well for individual users, but in a production setting the namespaces don't align quite as intuitively since each production deployment will get something like production:FlowName-<deployment #>-<hash>. This can be worked around by specifying the whole production token for the namespace, but curious what your all's thoughts are for this usage.

5 replies

joe153

@joe153

I am starting to get this error: [Errno 62] Too many levels of symbolic links when running a job with AWS batch on my mac. It worked fine before but just started today. Any ideas why? Only for environment=conda. Not obvious at the moment-
I still have the same python 3.8, same metaflow 2.2.3, same conda 4.8.4, etc.

8 replies

David Patschke

@dpatschke

Another question for the Netflix Metaflow team as I've searched and haven't found anything related ...
Is it possible to 'sync' the results/artifacts from a Run that exists within a local metadata store to the same Flow that exists on the remote metadata store on AWS?
I had a long-running Metaflow training Run which I executed locally and which had a significant amount of data artifacts that were being saved/tracked. I wasn't sure that the results of the Run were going to be exactly what I was desiring, but now the job has finished (several days later) and I've vetted the results, I would like to be able to 'merge' this Run with the Flow metadata that exists remotely.
Think of this almost as git-like functionality within the Metaflow metastore. I'm pretty sure the git-like behavior doesn't exist but perhaps there is a manual way of accomplishing this which I haven't run across?

10 replies

Revaapriyan

@Revaapriyan

Hi People. I would like to know that if there a way to limit the number of processor cores used by a metaflow code. In python's multiprocessing library I can specify the number of cores has to be used by the entire parallel processing tasks. But in metaflow I can specify a step or a task's minimum resources and could not find a way to restrict the amount of parallelism or how many times a step can be executed simultaneously.

2 replies

David Patschke

@dpatschke

Back again for another question for the Netflix Metaflow team. Metaflow does an awesome job of storing artifacts, allowing for tagging, and organizing of ML/DS experiments. On the flip side, I haven't really seen anything mentioned about actually managing the Flows themselves (other than inspection through the Python SDK). I'm visualizing/thinking about something along the lines of MLFlow's Model Registry. Just curious whether anything like this is leveraged internally and, perhaps, being considered for open source release or whether it is up to us mere mortals to come up with a solution ourselves. Would love any information you are able and willing to provide.

4 replies

russellbrooks

@russellbrooks

Hey guys, just wanted to point out a difference in retry behavior between batch jobs when using the local metaflow scheduler versus step functions. When jobs are retried normally with the metaflow scheduler, there's a 2min delay between attempts. When jobs are retried with step functions, the retries occur immediately. This appears to be from using the RetryStrategy at the batch job level, which doesn't support customization, whereas the step functions API allows for step-level retry logic that also supports delay intervals, exponential backoffs, etc. It may not be feasible to switch this over, but wanted to run it by you all and see if it's something you've considered.

4 replies

Greg Hilston

@GregHilston

Hey guys, I'm aware of the resources decorator for memory, CPU and GPU requests when running on AWS Batch but was wondering how Metaflow recommends handling the need of more disk space?

I've read that one can modify the AMI deployed on batch to get a larger than 8GB default volume size.

Is there a more friendly way to achieve this? I find myself working with datasets that are bigger than 8GB for some experiments but others use much less than 8GB.

Thanks!

7 replies

Andrew Achkar

@hajapy

Quick question/comment. We’re trying to package a number of flows into a standard python project structure. That is a top level package, modules and sub-packages. We cannot use absolute imports because when we launch the flow the import fails. So we have to go with imports that work at runtime, but our IDE doesn’t like. Is there a way to make this project structure work? How do you recommend structuring a project with multiple flows that share common utility modules/packages? My other comment is just that our IDE (pycharm) doesn’t recognize the decorators, which I think is due to them being dynamically created. Are there any workarounds for this?

13 replies

Greg Hilston

@GregHilston

Is there any documentation, procedure or scripts for transferring one Metdata service to another?

Imagine a user stood up all the infra in region A on AWS and wanted to move to region B without data loss.

I can write the S3 and postgres transfer scripts myself but was hoping to not re-invent the wheel.

Thanks!

Alireza Keshavarzi

@isohrab

Hi guys, I created a preprocessing Flow with some (default) parameters. It works well in my local machine and I able to run steps on the AWS. Now I want to integrate AWS Step functions to schedule my preprocessing and I created step function in the AWS with step-functions create command but when I execute manually from AWS console, I will get '$.Parameters' Error in AWS. Here is the whole message:

"An error occurred while executing the state 'start' (entered at the event id #2). The JSONPath '$.Parameters' specified for the field 'Value.$' could not be found in the input '{\n    \"Comment\": \"Insert your JSON here\"\n}'"

When I checked the state machine generated in AWS, I see that there is item in the Environment section as follow:

{
      "Name": "METAFLOW_PARAMETERS",
      "Value.$": "$.Parameters"
}

I checked following AWS resource : https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html
But I couldn't solve my problem. I believe I don't need to provide any parameters because I provided a default value for all parameters.
Do you have any idea? I appreciate your help.

3 replies

Greg Hilston

@GregHilston

Hey Metaflow, has anyone been able to create a single Batch Job Queue and Compute Environment that handles both CPU and GPU jobs, say with a p3.2xlarge?

I ask as I've seen others suggest online using two separate Job Queues, one for CPU and one for GPU jobs but Metaflow only supports a single Job Queue.

While my Compute Environment has successfully spun up p3.2xlarge instances, I have been unable to get a single GPU Step to leave the RUNNABLE state . I've been exploring if this is related to the AWS Launch Template I created to increase the disk size of my instances.

If anyone has any advice, documentation or examples of running GPU jobs along side CPU jobs in the same Batch Job Queue and Compute Environment with Metaflow, I'd very much appreciate it

2 replies

russellbrooks

@russellbrooks

Hey guys, nearly forgot to follow up and share some advice when creating Batch compute environments that was especially relevant towards my previous issues when having SFN-executed, highly parallel and short-running jobs being co-located on large instances:

Not using the default Batch ECS-optimized AMIs that are still using the soon-to-be deprecated Amazon Linux 1 AMIs instead of the latest ECS-optimized Amazon Linux 2 AMIs.

The Linux 1 AMI uses the Docker devicemapper storage driver, and preallocates 10GB of per-container storage. The Linux 2 AMIs use the Docker overlay2 storage driver, which exposes all unused space on the disk to running containers. 

Manually setting my Batch compute environments to use the latest ECS-optimized Linux 2 AMIs seems to be the cleanest approach, rather than playing with custom ECS Agent docker cleanup parameters. I also reached out to AWS support to see if there’s a reason why Batch hasn’t updated their default AMI, even though the Linux 1 AMI is end-of-life in 2 months. No information was given, but mentioned that they have an internal feature request for it without any guarantees or ETA on when it’d be changed.

Sharing in case this is useful for anyone else!

Where communities thrive

People

Repo info

Activity