6 replies
https://github.com/Netflix/metaflow/blob/master/metaflow/tutorials/08-autopilot/autopilot.ipynb
I created a step function for my flow and specified CPU and RAM for steps using resources decorator, but after the run I noticed that it ignores values (to be more correct, it runs on default container) that are lower than metaflow defaults which are cpu : 1, memory : 4000. The situation is different when using batch decorator, in that case it allows smaller containers. Anyone facing this issue or having an idea why we can't specify smaller resources while running flow as a step function ?
7 replies
I am trying to run metaflow inside a docker container, but I have to run it as non-root user. When I try to import with
"metaflow configure import metaflow_config/config.txt"
I get "PermissionError: [Errno 13] Permission denied: '/.metaflowconfig'"
I have tried changing the permissions with chown and chmod, currently set to
drwxrwxrwx 2 1000 1000 6 Apr 5 19:01 .metaflowconfig
But no luck. Can I run metaflow inside a docker container without being a root user?
6 replies
4 replies
20 replies
Separately, anyone had this issue?
I can run aws c3 cp (my file) (remote bucket) and it pulls from the $AWS_PROFILE variable correctly (with an alias set on aws command to do so). However, running METAFLOW_PROFILE=personal python 05-helloaws/helloaws.py --datastore=s3 run I'm getting a token expired error which, I think is because it is using the wrong profile. Any tips on how to debug without just switching profile names. I will need to use a named profile.
7 replies
I've created an Netflix/metaflow#473 to stop supporting Python 2.x from the next Metaflow release. Mostly because it will allow type annotations from Python 3, and make codebase more contributor friendly.
It seems like a pretty conservative move given that Python 2.7 has been EOL'ed more than a year ago. But I'm curious if anyone here is still using Metaflow with Python 2.7 and would be affected by this change?
I am getting internal server errors when I add 'METAFLOW_DEFAULT_METADATA': 'service' to my config file. My config file contains METAFLOW_SERVICE_URL, METAFLOW_SERVICE_INTERNAL_URL and METAFLOW_SERVICE_AUTH_KEY and I have verified they match what is in the cloudformation stack output.
when I try to run a script locally with
python inference-flow.py --environment=conda --datastore=s3 run
I get the following
Bootstrapping conda environment...(this could take a few minutes)
Metaflow service error:
Metadata request (/flows/inference-flow) failed (code 500): {"message": "Internal server error"}If I try to run step functions create I get the following:
Running pylint...
Pylint is happy!
Deploying inference_flow to AWS Step Functions...
Internal error
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/metaflow/cli.py", line 930, in main
start(auto_envvar_prefix='METAFLOW', obj=state)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/aws/step_functions/step_functions_cli.py", line 88, in create
check_metadata_service_version(obj)
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/aws/step_functions/step_functions_cli.py", line 120, in check_metadata_service_version
version = metadata.version()
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/metadata/service.py", line 41, in version
return self._version(self._monitor)
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/metadata/service.py", line 288, in _version
(path, resp.status_code, resp.text),
NameError: name 'path' is not definedlist(Metaflow()) gives the following
Traceback (most recent call last):
File "cleanup.py", line 14, in <module>
print('list(Metaflow())',list(Metaflow()))
File "/usr/local/lib/python3.7/site-packages/metaflow/client/core.py", line 245, in __iter__
all_flows = self.metadata.get_object('root', 'flow')
File "/usr/local/lib/python3.7/site-packages/metaflow/metadata/metadata.py", line 357, in get_object
return cls._get_object_internal(obj_type, type_order, sub_type, sub_order, filters, *args)
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/metadata/service.py", line 116, in _get_object_internal
return MetadataProvider._apply_filter(cls._request(None, url), filters)
File "/usr/local/lib/python3.7/site-packages/metaflow/plugins/metadata/service.py", line 247, in _request
resp.text)
metaflow.plugins.metadata.service.ServiceException: Metadata request (/flows) failed (code 500): {"message": "Internal server error"}
script returned exit code 1Any ideas what might be happening?
16 replies
self.next ?
3 replies
if x:
self.next(...)
else
self.next(...)
Hi, question about isolation of steps, I noticed that these two steps will be executed in the same conda environment:
from metaflow import FlowSpec, step, conda
class IsolationTest(FlowSpec):
@conda(python="3.8.5")
@step
def start(self):
import sys
print(f"start executable: {sys.executable}")
self.next(self.end)
@conda(python="3.8.5")
@step
def end(self):
import sys
print(f"end executable: {sys.executable}")
if __name__ == "__main__":
IsolationTest()They will run in separated environments only if the python version is different. Is there a way to ensure that separate environments are created? (context: I need to install packages from pip, which results in heavy installing / rolling back packages every time flow is executed)
5 replies
2 replies
We are considering porting our data processing/ML pipelines to metaflow, but one thing that is holding us back is the lack of support for integration testing. For instance, we currently use kedro for our data pipelines. Kedro provides the ability to call and run a kedro pipeline from another python script (directly, without using subprocess.run) and additionally provides the ability to override the default datasets used in that pipeline at runtime. This is important, because some of our datasets are very large, so we naturally want to use subsetted versions of these datasets. In fact, some of the datasets we inject into the pipeline for the integration test are generated with Hypothesis which ensures that our pipelines are robust to unanticipated variations in the data.
Furthermore, although it's not ideal for an integration test, there are some expensive functions, or functions that rely on a network connection, that we want to patch using unnitest.mock.patch . This doesn't seem to be possible when running a metaflow pipeline with subprocess.run .
One solution could be to just to have a test flow inherit from the flow to be tested, and override the artifacts that way, ie:
class TestPlaylistFlow(PlayListFlow):
movie_data = test_dataBut as far as I understand it, the child flow will not inherit the step functions, which would force us to import them and manually specify them in the test flow.
The only other solution I can think of at present is to define a boolean parameter test in the original flow and based on the value of that parameter assign different artifacts to the instance variables as necessary. Is there another option that can be implemented with the current version of metaflow==2.2.9?
11 replies
hey guys, had a teammate run into what I believe is a bug in how Parameters are used in Step Functions deployments. If there is a Parameter that defaults to None like
Parameter(name="test_param", type=int, default=None)it'll result in the following error when deploying with step-functions create.
Flow failed:
The value of parameter test_param is ambiguous. It does not have a default and it is not required.The same flow/parameter will run successfully locally or when submitted to batch without SFNs.
1 reply
the auto increment number is fine, but I'd like to use a name ideally when kicking off jobs from the command line so it's easier to keep track of results from multiple runs and compare them easily
2 replies
1 reply
@conda(...) decorators) is making it quite difficult when relying on pip-installed packages - if the step which requires pip package is executed in parallel within foreach then pip will fail as multiple threads will be trying to install packages into the same environment. I'm thinking about possible solutions - maybe a switch to force creation of unique conda environments for each step / option to run each step in separate docker container / proper support for pip through @pip. What do you think? I'd be happy to contribute
7 replies
1 reply
The use case is we have two activities where the second depends on the first, but where it doesn't necessarily make sense to split them into separate steps and the first activity takes a while to complete. During development, say we are modifying the second activity, then every time we run another test of our changes, we have to execute that first long activity again. In order to speed things up during dev, we might split up into separate steps to allow resuming at the right place, but then we might combine them into a single step for higher efficiency in production
3 replies
@tuulos @savingoyal I'm seeing some strange behavior when running Flows via AWS Batch that are not consistently reproducible. The error I'm getting says that:
Task is starting.
list index out of range
Task failed.There is no hash present within hard brackets via the Metaflow logging so something is telling me this might be an undetected AWS Batch issue ( but I have no idea). It's like the Task never even got a chance to start and hard-failed from there.
What really stinks about this error is that the Flow is run with --with retry but a retry is not even attempted. This is the 2nd time this error has presented itself to me within a week. Both times I have been able to re-run the Flow immediately afterwards, and it completes successfully. FWIW, this last time, it happened in the middle of a foreach fan-out and all other fanned out processes kept running. The Flow failed at the end step because, I'm guessing all known tasks didn't complete successfully and a check is made (good work on your end for having that).
Are either of you aware of what may be causing this error and/or have any potential suggestions?
Thanks!
9 replies
python my_flow.py run --namespace test --tag test, the last_run of the flow in the user namespace is updated with the run in the test namespace. Furthermore, It doesn't seem to be possible to run flows as a different user using the --namespace flag:python my_flow.py run --namespace user:test
...
2021-04-29 18:18:04.147 [1619709482612481/end/5 (pid 67190)] Task finished successfully.
namespace("user:daavid")
Metaflow().flows[0].latest_run
>> Run('PlayListFlow/1619709482612481')
Metaflow().flows[0].latest_run.tags
>> frozenset({'date:2021-04-29',
'metaflow_version:2.2.10',
'python_version:3.8.5',
'runtime:dev',
'user:daavid'})
19 replies
Hello, I've recently installed metaflow on AWS using a manual deployment. I'm running the helloaws.py flow and it's stuck in the RUNNING state. Looking at the log steam I see the code is downloaded but again it's just stuck at Task is starting.
Can someone please help me diagnose the problem?
4 replies
https://ayotomiwasalau.medium.com/starting-your-data-science-project-with-metaflow-the-mnist-use-case-44e3b3ad6ec3
5 replies
5 replies
Hi all, taking some ideas from the community here. Would like you all to “roast” my potential ML platform Arch.
The purpose is to provide a centralized place to handle the ML Training piplines. lThis is specific to using Metaflow
1.Users executes their Metaflow “jobs”
2.Metaflow runs the training pipeline jobs
3.Outputs a model packaged as a docker image, stored in AWS ECR
4.Gets duplicate/pulled into users of different teams/business accounts.
Anyone had this approach before ? or i’m doing it wrong
