Hi, question about isolation of steps, I noticed that these two steps will be executed in the same conda environment:
from metaflow import FlowSpec, step, conda
class IsolationTest(FlowSpec):
@conda(python="3.8.5")
@step
def start(self):
import sys
print(f"start executable: {sys.executable}")
self.next(self.end)
@conda(python="3.8.5")
@step
def end(self):
import sys
print(f"end executable: {sys.executable}")
if __name__ == "__main__":
IsolationTest()They will run in separated environments only if the python version is different. Is there a way to ensure that separate environments are created? (context: I need to install packages from pip, which results in heavy installing / rolling back packages every time flow is executed)
5 replies
2 replies
We are considering porting our data processing/ML pipelines to metaflow, but one thing that is holding us back is the lack of support for integration testing. For instance, we currently use kedro for our data pipelines. Kedro provides the ability to call and run a kedro pipeline from another python script (directly, without using subprocess.run) and additionally provides the ability to override the default datasets used in that pipeline at runtime. This is important, because some of our datasets are very large, so we naturally want to use subsetted versions of these datasets. In fact, some of the datasets we inject into the pipeline for the integration test are generated with Hypothesis which ensures that our pipelines are robust to unanticipated variations in the data.
Furthermore, although it's not ideal for an integration test, there are some expensive functions, or functions that rely on a network connection, that we want to patch using unnitest.mock.patch . This doesn't seem to be possible when running a metaflow pipeline with subprocess.run .
One solution could be to just to have a test flow inherit from the flow to be tested, and override the artifacts that way, ie:
class TestPlaylistFlow(PlayListFlow):
movie_data = test_dataBut as far as I understand it, the child flow will not inherit the step functions, which would force us to import them and manually specify them in the test flow.
The only other solution I can think of at present is to define a boolean parameter test in the original flow and based on the value of that parameter assign different artifacts to the instance variables as necessary. Is there another option that can be implemented with the current version of metaflow==2.2.9?
15 replies
hey guys, had a teammate run into what I believe is a bug in how Parameters are used in Step Functions deployments. If there is a Parameter that defaults to None like
Parameter(name="test_param", type=int, default=None)it'll result in the following error when deploying with step-functions create.
Flow failed:
The value of parameter test_param is ambiguous. It does not have a default and it is not required.The same flow/parameter will run successfully locally or when submitted to batch without SFNs.
1 reply
the auto increment number is fine, but I'd like to use a name ideally when kicking off jobs from the command line so it's easier to keep track of results from multiple runs and compare them easily
2 replies
1 reply
@conda(...) decorators) is making it quite difficult when relying on pip-installed packages - if the step which requires pip package is executed in parallel within foreach then pip will fail as multiple threads will be trying to install packages into the same environment. I'm thinking about possible solutions - maybe a switch to force creation of unique conda environments for each step / option to run each step in separate docker container / proper support for pip through @pip. What do you think? I'd be happy to contribute
7 replies
1 reply
The use case is we have two activities where the second depends on the first, but where it doesn't necessarily make sense to split them into separate steps and the first activity takes a while to complete. During development, say we are modifying the second activity, then every time we run another test of our changes, we have to execute that first long activity again. In order to speed things up during dev, we might split up into separate steps to allow resuming at the right place, but then we might combine them into a single step for higher efficiency in production
3 replies
@tuulos @savingoyal I'm seeing some strange behavior when running Flows via AWS Batch that are not consistently reproducible. The error I'm getting says that:
Task is starting.
list index out of range
Task failed.There is no hash present within hard brackets via the Metaflow logging so something is telling me this might be an undetected AWS Batch issue ( but I have no idea). It's like the Task never even got a chance to start and hard-failed from there.
What really stinks about this error is that the Flow is run with --with retry but a retry is not even attempted. This is the 2nd time this error has presented itself to me within a week. Both times I have been able to re-run the Flow immediately afterwards, and it completes successfully. FWIW, this last time, it happened in the middle of a foreach fan-out and all other fanned out processes kept running. The Flow failed at the end step because, I'm guessing all known tasks didn't complete successfully and a check is made (good work on your end for having that).
Are either of you aware of what may be causing this error and/or have any potential suggestions?
Thanks!
9 replies
python my_flow.py run --namespace test --tag test, the last_run of the flow in the user namespace is updated with the run in the test namespace. Furthermore, It doesn't seem to be possible to run flows as a different user using the --namespace flag:python my_flow.py run --namespace user:test
...
2021-04-29 18:18:04.147 [1619709482612481/end/5 (pid 67190)] Task finished successfully.
namespace("user:daavid")
Metaflow().flows[0].latest_run
>> Run('PlayListFlow/1619709482612481')
Metaflow().flows[0].latest_run.tags
>> frozenset({'date:2021-04-29',
'metaflow_version:2.2.10',
'python_version:3.8.5',
'runtime:dev',
'user:daavid'})
20 replies
Hello, I've recently installed metaflow on AWS using a manual deployment. I'm running the helloaws.py flow and it's stuck in the RUNNING state. Looking at the log steam I see the code is downloaded but again it's just stuck at Task is starting.
Can someone please help me diagnose the problem?
4 replies
https://ayotomiwasalau.medium.com/starting-your-data-science-project-with-metaflow-the-mnist-use-case-44e3b3ad6ec3
5 replies
7 replies
Hi all, taking some ideas from the community here. Would like you all to “roast” my potential ML platform Arch.
The purpose is to provide a centralized place to handle the ML Training piplines. lThis is specific to using Metaflow
1.Users executes their Metaflow “jobs”
2.Metaflow runs the training pipeline jobs
3.Outputs a model packaged as a docker image, stored in AWS ECR
4.Gets duplicate/pulled into users of different teams/business accounts.
Anyone had this approach before ? or i’m doing it wrong
13 replies
Hi guys, having trouble with conda dependencies when trying to run a flow on AWS batch. I'm working on an M1 Macbook with Anaconda. My workflow was to develop as usual in a local conda env, then port my main.py script to a Metaflow graph with conda_base decorator, listing the package versions as installed in the local env:
@conda_base(libraries={'numpy':'1.19.5',
"pyyaml":"5.4.1" ,
"torchvision":"0.9.0",
"requests": "2.25.1",
"pytorch": "1.8.0"
},
python='3.7')
This Metaflow script ran perfectly locally, successfully bootstrapping the conda environment and running the flow. When running with "--with batch" the conda build failed and printed a ton of package clashes (see above). I havent been able to resolve these but i can successfully run on batch with a simple python 3.7 + numpy conda env. Is there something up with our AWS setup that is causing these packages which evidently are compatible, to clash on batch?
Hi all, we are investigating Metaflow coming from a DVC background. What we have seen so far with Metaflow is great, in particular the seamless deployment to AWS is a huge benefit over DVC. However, I am missing some features that DVC had in relation to transparency of runs.
For instance, is there a way to retrieve all parameter artifacts from a particular run for easy comparison of parameters between runs? I see that parameters are stored as artifacts, but programmatically they are mixed in with all other artifacts.
9 replies
17 replies
Hi all, am going through the tutorial 4 of metaflow https://github.com/Netflix/metaflow/blob/master/metaflow/tutorials/04-playlist-plus/playlist.py, when running it with —environment conda —with batch
i’m getting the following in the aws batch logs
bash: metaflow_PlayListFlow_linux-64_a55a81bc46d543afe1d885c81703deeaae95efef/bin/python: No such file or directorythis is running the default python:3.8 image.
any thoughts what i should be doing to fix this ?
24 replies
3 replies
Do you guys trigger metaflow runs in production from API gateway. I'd like to effectively use my metaflow flow to back an inference endpoint, but that might be stupid. I'm not sure. What's the right way to go about this?
I have a metaflow flow for inference (it's a multi-model setup and a complicated inference pipeline which is why this works best). I normally trigger it from the CLI, but now I want to ship it to production
1 reply
This release introduces a new decorator @project for managing/coordinating larger Metaflow projects besides other exciting features. Now, Metaflow users can iterate on & deploy experimental versions of their workflows alongside production deployments in a safe & isolated manner.
Is there any way to specify common commands to be run before every step in a flow? Perhaps a sort of pre-step function?
My use case is trying to integrate mlflow: I need to make calls such as mlflow.start_run(run_id=...) at the start of every step to ensure all steps log to the same mlflow run.
Additionally, any other tips surrounding use of metaflow and mlflow together would be very helpful.
4 replies
Hi!
Is there some reference to using minio as a datastore?
I tried setting the variables like:
METAFLOW_S3_ENDPOINT_URL=http://adm:pswd@xx.x.x.x:PORT METAFLOW_DATATOOLS_SYSROOT_S3=/minio/metaflow-dir/data METAFLOW_DATASTORE_SYSROOT_S3=/minio/metaflow-dir python mf.py runBut I get "invalid bucket name" error.
6 replies
METAFLOW_JOB_QUEUE_NAME python parameter_flow.py step-functions trigger --alpha 0.5
5 replies
📣 hey all - @savingoyal, @oavdeev and I founded a startup, Outerbounds, to help the wider Metaflow community outside Netflix. We have a new Slack instance at http://slack.outerbounds.co that has a good number of companies and individuals using Metaflow. You are welcome to join it! 👋
We will keep monitoring this Gitter channel too but many people seem to prefer the ergonomics of Slack over Gitter.