Hey guys, first of all thank you for open sourcing Metaflow, it seems great so far!
I've been working on setting up CodePipelines for Metaflow, to execute flows for our engineers automatically. This work well for the Hello world flows, but fails to run when I execute this step
@batch(cpu=1, memory=500)
@step
def hello(self):
"""
Time to run some complicated operations on AWS batch!
"""
self.message = 'Hi from AWS Batch!'
print("Metaflow says: %s" % self.message)
self.next(self.end)With the following error:
2020-02-17 08:48:50.596 [1581929268823091/hello/2 (pid 176)] [518aa708-3bd6-4cb3-b821-b1b028da6248] botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints
2020-02-17 08:48:50.597 [1581929268823091/hello/2 (pid 176)] Batch error:
2020-02-17 08:48:50.666 [1581929268823091/hello/2 (pid 176)] Task crashed. This could be a transient error. Use @retry to retry.Did anyone experience something similar? The same flow works great locally. Why could it be that boto3 within CodeBuild cannot access the endpoints?
Thanks @savingoyal . I was able to run with a local metadata provider.
I had a one more question. I was writing a Kubernetes plugin that will work similarly to the Batch plugin. Now inorder to understand the functionality of the Batch plugin better, I wanted to export my own version of Metaflow when exporting the environment via Conda to S3. So that when It runs on Batch, It will import my version of metaflow and help me debug the internal functioning of the library. Can you suggest a way I can test?
I was also facing an error in run after I deleted my Conda env folder and reinstalled conda. Should I open a PR to handle this error a little more gracefully? Everything worked fine after I cleared conda.dependencies file.
When I was running metaflow with batch, I noticed that the full package of deps gets downloaded using conda for each of the step nodes.
I saw this here : "python -m metaflow.plugins.conda.batch_bootstrap \"%s\" %s" % \
(self.flow.name, env_id), in metaflow/metaflow/plugins/conda/conda_environment.py
Is there any advantage of defining conda based dependencies in the conda_base decorator over the conda step decorator?
Because bootstrapping environment each time for heavy deps like tensorflow is taking too long each time the step starts on AWS.
2020-02-17 21:31:33.881 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Task is starting (status RUNNING)...
2020-02-17 21:31:40.188 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Hello This is a Hi From Custom Library
2020-02-17 21:31:40.189 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Setting up task environment.
2020-02-17 21:31:54.224 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Downloading code package.
2020-02-17 21:31:54.225 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Code package downloaded.
2020-02-17 21:32:26.636 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Bootstrapping environment.
2020-02-17 21:32:26.636 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Environment bootstrapped.
2020-02-17 21:32:33.918 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Task is starting.I had a question regarding the DATASTORE_SYSROOT_S3 environment variable.
I was trying to isolate and run each of these flows in a seperate container on my machine. Below is a command snipped which is run inside the batch container.
'metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582000148684507 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}'The container I spawned with the above command failed with error.Data store error:
Datastore root not found. Specify with DATASTORE_SYSROOT_S3 environment variable.
What is the different between DATASTORE_SYSROOT_S3 and --datastore-root provided in the command.
@leftys : " it seems it's not possible to run the whole flow in the same python interpreter": Yes. the Nativeruntime will spawn processes for steps/tasks in the graph and those processes invoke every step with the conda based dependencies/python/data coming from the Datastore.
For Example
['/Users/valaydave/Documents/Code-Workspace/Exploration-Projects/Metaflow_Local_Test/.env/bin/python', 'hello.py', '--quiet', '--metadata', 'local', '--environment', 'conda', '--datastore', 's3', '--event-logger', 'nullSidecarLogger', '--monitor', 'nullSidecarMonitor', '--datastore-root', 's3://kubernetes-first-test-store/metaflow_store', 'batch', 'step', 'start', 'db1377d8ac93cf9acd9898258349fc1720f60f8a', 's3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db1377d8ac93cf9acd9898258349fc1720f60f8a', '--run-id', '1582000148684507', '--task-id', '1', '--input-paths', '1582000148684507/_parameters/0', '--cpu', '2', '--gpu', '0', '--memory', '2000', '--image', 'python:3.7', '--queue', 'arn:aws:batch:us-east-1:123456789535:job-queue/Metaflow-Job-Q', '--iam_role', 'arn:aws:iam::123456789535:role/Metaflow_Batch_ECS_Role_TEST', '--run-time-limit', '432000']The above are args it uses to call the step in a subprocess which will package the Job for batch for execution . Once on batch it runs something like this to invoke the processsing :
'metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582000148684507 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}'Unit test for step should ideally work with persistent data in the datastore. The other thing I am assuming, is that you have set conda as environment manager. If conda is set, then you every subprocess which is being run, is having the same env as the other subprocesses so ideally any test run locally should work if packaged correctly for remote.
I hope this makes sense. I am answering based on intuition and my understanding of metaflow. Please do let me know if you think I am incorrect somewhere.
@valayDave Unit tests should not run on some real data, but on artificially created data or several sets of data depending on the test case. So I need to mock the data, ideally together with the testing code for good readability/maintanability.
I see some way to perform unit-testing: I would just run the single step method separately in the same process and mock input and output.
For integration testing, the best I could think of is to run the flow externally by running a command, use data from datastore (thats acceptable for me for integration testing) and check results from local metadata store.
@leftys : "So I need to mock the data, ideally together with the testing code for good readability/maintanability." Exactly this. But In a linear flow, each step is will "Create" data for the next step. Now in such a situation, I am assuming that you will have to mock the data created by the flow at each step to unit test the step?
Are you planning to use pytest for unit testing? If you are can u share some specs on how u will test individual steps for the flow? I also wanted to know if you have any strategy regarding autogenerating the mock data for each step
@savingoyal : What do u mean by "How are you initializing the datastore?". I have my current metaflow provider configured by metaflow configure aws. I have set S3 as the datastore and everything works fine with Batch. I am trying to understand the importance of the --data-root command-line argument.
I was running my container-based isolated step without the environment variables of METAFLOW_DATASTORE_SYSROOT_S3. I was assumed that providing --datastore-root is the same as METAFLOW_DATASTORE_SYSROOT_S3 env var. If I ran with all env vars of that batch sets then the container-based isolated step works completely fine.
Yes, I actually did run into this error while developing a k8s plugin. I was trying to write a mock pod spec with which I would be able to run the job.
apiVersion: batch/v1
kind: Job
metadata:
name: metaflow-runner
spec:
template:
spec:
containers:
- name: testing-import
image: python:3.7
env:
- name: AWS_ACCESS_KEY_ID
value: AAAAAAAAAAAAAAAAAAAA
- name: AWS_SECRET_ACCESS_KEY
value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
- name: AWS_SESSION_TOKEN
value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
- name: METAFLOW_CODE_SHA
value: db02ca64a7b31d0abcefbd36b7858947989a7ac4
- name: METAFLOW_CODE_URL
value: s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4
- name: METAFLOW_CODE_DS
value: s3
- name: METAFLOW_USER
value: valaydave
- name: METAFLOW_SERVICE_URL
value: None
- name: METAFLOW_SERVICE_HEADERS
value: "{}"
- name: METAFLOW_DATASTORE_SYSROOT_LOCAL
value: ".metaflow"
- name: METAFLOW_DATATOOLS_S3ROOT
value: s3://kubernetes-first-test-store/metaflow_store/data
- name: METAFLOW_DEFAULT_DATASTORE
value: s3
- name: METAFLOW_DEFAULT_METADATA
value: local
- name: METAFLOW_INPUT_PATHS_0
value: 1582048815350055/_parameters/0
command: ["sh"]
args: ["-c",
"set -e && echo 'Hello This is a Hi From Custom Library' && echo 'Setting up task environment.' && python -m pip install awscli click requests boto3 --user -qqq && mkdir metaflow && cd metaflow && i=0; while [ $i -le 5 ]; do echo 'Downloading code package.'; python -m awscli s3 cp s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4 job.tar >/dev/null && echo 'Code package downloaded.' && break; sleep 10; i=$((i+1));done && tar xf job.tar && echo 'Bootstrapping environment.' && python -m metaflow.plugins.conda.batch_bootstrap HelloAWSFlow metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83 && echo 'Environment bootstrapped.' && echo 'Task is starting.' && metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582048815350055 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}"
]
restartPolicy: Never
backoffLimit: 4The above is a POD template that doesn't work because it doesn't have the METAFLOW_DATASTORE_SYSROOT_S3 env var.
It give me this error :
Hello This is a Hi From Custom Library
Setting up task environment.
Downloading code package.
Code package downloaded.
Bootstrapping environment.
Environment bootstrapped.
Task is starting.
Data store error:
Datastore root not found. Specify with DATASTORE_SYSROOT_S3 environment variable.But putting the METAFLOW_DATASTORE_SYSROOT_S3 env var, I works fine:
Hello This is a Hi From Custom LibrarySetting up task environment.Downloading code package.Code package downloaded.
Bootstrapping environment.
Environment bootstrapped.
Task is starting.
Hellow From AWSI just wanted to know that do I need to always specify the env var even after putting command line args ?
