Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Valay Dave
    @valayDave
    Yes. I am building a Datapipeline which requires data to be fetched daily for creating models. So I was thinking of using Metaflow to craft this pipeline. If there is a way automatically schedule metaflow to run, the Metaflow will very nicely with my usecase
    Ville Tuulos
    @tuulos
    yep, @savingoyal is working on integration between Metaflow and AWS Step Functions, if using AWS is an option for you. If you just need something simple on a local machine, technically you could just use cron
    Jonathan Streater
    @prometheusred
    Thank you all for Metaflow, I've just set it up on my company's AWS account using your cloudformation script to start experimenting with it. I'm excited about the metaflow philosophy. And I'm extremely excited to keep up with things on github. But anyway, I have another question for you on the topic of using metaflow in the context of a live data pipeline (as opposed to just within the dev/modeling loop). Obviously, I'm excited to see that there might be a PR soon on integrating metaflow with Step Functions. But I was wondering if you had any recommendations about using metaflow within this live pipeline context in the mean time. In the next two weeks i have the opportunity to replicate a few data pipelines used for preprocessing, inference, and (re)training that are currently done with some lambdas, ec2, and cloudwatch events. My ultimate goal is to focus on and experiment with the human-centered philosophy and workflow espoused by Metaflow that could enable more effective iteration and experimentation within the context of my company's tooling (or lack thereof ). However, to even get that conversation going, I need to get this thing working within that live pipeline context, working with larger than will fit into memory data, first. This is a vague ask but if you've got any pointers or direction, I'd appreciate it. Would it be reasonable to just kick flows off from within a small ec2 or lambda triggered by s3 events?
    Ville Tuulos
    @tuulos
    hi @prometheusred ! Good to hear that the philosophy is resonating with you. What are your requirements for the "live pipeline" use case? Is it about low-latency streaming of data? When you say "larger data than what fits into memory", what's the scale we're talking about?
    Ferdinand von den Eichen
    @fvde

    Hey guys, first of all thank you for open sourcing Metaflow, it seems great so far!

    I've been working on setting up CodePipelines for Metaflow, to execute flows for our engineers automatically. This work well for the Hello world flows, but fails to run when I execute this step

    @batch(cpu=1, memory=500)
        @step
        def hello(self):
            """
            Time to run some complicated operations on AWS batch!
    
            """
            self.message = 'Hi from AWS Batch!'
            print("Metaflow says: %s" % self.message)
            self.next(self.end)

    With the following error:

    2020-02-17 08:48:50.596 [1581929268823091/hello/2 (pid 176)] [518aa708-3bd6-4cb3-b821-b1b028da6248] botocore.exceptions.DataNotFoundError: Unable to load data for: endpoints
    2020-02-17 08:48:50.597 [1581929268823091/hello/2 (pid 176)] Batch error:
    2020-02-17 08:48:50.666 [1581929268823091/hello/2 (pid 176)] Task crashed. This could be a transient error. Use @retry to retry.

    Did anyone experience something similar? The same flow works great locally. Why could it be that boto3 within CodeBuild cannot access the endpoints?

    Ferdinand von den Eichen
    @fvde
    NVM - I resolved it myself :)
    Jan Škoda
    @leftys
    @prometheusred Maybe I have a similar use case, processing real-time market data. And I think Metaflow design/philosophy doesn't match this use-case. Our plan is to use Metaflow just for batch processing and training models and implement streaming pipelines separately (with as much shared code as possible).
    Jonathan Streater
    @prometheusred
    @leftys definitely an interesting issue about how/if the human centered philosophy supports not just the iterate offline data science workflow but also the shift/transition between deploying and experimenting with online systems. I'm definitely thinking a lot about this at the moment. In my case, I feel metaflow (perhaps in combination with sagemaker) might work, especially since I'm just prototyping at the moment with pipelines that are not real-time but I also don't want to rule out things like sagemaker or MLFlow just yet either.
    Yosi Taguri
    @yosit
    Hi, is anyone working on a UI service for metaflow?
    Jonathan Streater
    @prometheusred
    @tuulos it is not low-latency but rather daily batch processing which might not actually be complicated/large memory-wise (i'll go verify) with some occasional/ad-hoc processing of larger spans of time (several months) that might be in the 100's of GB (I'll go verify that as well, though now that you've asked me, I'm thinking maybe it's not that much either)
    maybe "automated" pipeline would have been better phrasing than "live" pipeline
    Ville Tuulos
    @tuulos
    @prometheusred that sounds pretty close to how we use Metaflow internally. @savingoyal is working on an integration with AWS Step Functions, which will enabled automated (i.e. scheduled / triggered) pipelines. Stay tuned :smiley:
    @yosit not yet but it would be great if someone with enough frontend-fu could start hacking one
    Valay Dave
    @valayDave
    I am trying to use Metaflow with AWS Resources. I had the following question :
    • I created Batch Queue, Batch Env, ECS Service Role and S3 Bucket needed for storing computed data. Can I keep S3/Local as a metadata provider?
    Savin
    @savingoyal
    You would need to have S3 as the datastore to execute on AWS Batch. You can have local as the metadata provider.
    Valay Dave
    @valayDave

    Thanks @savingoyal . I was able to run with a local metadata provider.

    I had a one more question. I was writing a Kubernetes plugin that will work similarly to the Batch plugin. Now inorder to understand the functionality of the Batch plugin better, I wanted to export my own version of Metaflow when exporting the environment via Conda to S3. So that when It runs on Batch, It will import my version of metaflow and help me debug the internal functioning of the library. Can you suggest a way I can test?

    I was also facing an error in run after I deleted my Conda env folder and reinstalled conda. Should I open a PR to handle this error a little more gracefully? Everything worked fine after I cleared conda.dependencies file.

    Savin
    @savingoyal
    @valayDave Yes a PR would be much appreciated.
    On AWS Batch, we already use the metaflow that launched the flow. You can take a look at package.py about how we package the code and deploy to AWS Batch using S3
    So if you pip install your own version of metaflow (pip install -e .), that would be the metaflow that will get deployed on AWS Batch.
    Valay Dave
    @valayDave
    @savingoyal : Thank you. This is very useful.
    Valay Dave
    @valayDave

    When I was running metaflow with batch, I noticed that the full package of deps gets downloaded using conda for each of the step nodes.
    I saw this here : "python -m metaflow.plugins.conda.batch_bootstrap \"%s\" %s" % \ (self.flow.name, env_id), in metaflow/metaflow/plugins/conda/conda_environment.py

    Is there any advantage of defining conda based dependencies in the conda_base decorator over the conda step decorator?

    Because bootstrapping environment each time for heavy deps like tensorflow is taking too long each time the step starts on AWS.

    Savin
    @savingoyal
    The env_id is specific to the step. If you specify dependencies using ‘conda_base’, those dependencies are available to every step while ‘conda’ makes dependencies available to only the specific steps.
    Ville Tuulos
    @tuulos
    @valayDave "is taking too long" - what's the delay you are observing? We have plans to reduce the latency
    Valay Dave
    @valayDave
    It took 1 Minute from when the task started.
    2020-02-17 21:31:33.881 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Task is starting (status RUNNING)...
    2020-02-17 21:31:40.188 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Hello This is a Hi From Custom Library
    2020-02-17 21:31:40.189 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Setting up task environment.
    2020-02-17 21:31:54.224 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Downloading code package.
    2020-02-17 21:31:54.225 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Code package downloaded.
    2020-02-17 21:32:26.636 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Bootstrapping environment.
    2020-02-17 21:32:26.636 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Environment bootstrapped.
    2020-02-17 21:32:33.918 [1582000148684507/start/1 (pid 17292)] [0921f520-2fa3-4745-99a6-b0bf25d9f484] Task is starting.
    I was running 2 cpu 2 GB instance.
    Ville Tuulos
    @tuulos
    thanks - seems to be in line with what we have seen before
    Valay Dave
    @valayDave

    I had a question regarding the DATASTORE_SYSROOT_S3 environment variable.

    I was trying to isolate and run each of these flows in a seperate container on my machine. Below is a command snipped which is run inside the batch container.

    'metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582000148684507 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}'

    The container I spawned with the above command failed with error.
    Data store error: Datastore root not found. Specify with DATASTORE_SYSROOT_S3 environment variable.

    What is the different between DATASTORE_SYSROOT_S3 and --datastore-root provided in the command.

    Jan Škoda
    @leftys
    @valayDave Kubernetes plugin sounds cool! We are thinking about using Metaflow with Kubernetes too.
    I am trying to write tests for my flow now -- unittests for individual steps and then integration test for the whole flow. But it seems it's not possible to run the whole flow in the same python interpreter (so that i can for example mock inputs for the integration test). Does anyone have an idea how to do this? So far I though about implementing some MockFlow class that would run in a single process, execute the graph without interacting with data store.
    Valay Dave
    @valayDave

    @leftys : " it seems it's not possible to run the whole flow in the same python interpreter": Yes. the Nativeruntime will spawn processes for steps/tasks in the graph and those processes invoke every step with the conda based dependencies/python/data coming from the Datastore.

    For Example

    ['/Users/valaydave/Documents/Code-Workspace/Exploration-Projects/Metaflow_Local_Test/.env/bin/python', 'hello.py', '--quiet', '--metadata', 'local', '--environment', 'conda', '--datastore', 's3', '--event-logger', 'nullSidecarLogger', '--monitor', 'nullSidecarMonitor', '--datastore-root', 's3://kubernetes-first-test-store/metaflow_store', 'batch', 'step', 'start', 'db1377d8ac93cf9acd9898258349fc1720f60f8a', 's3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db1377d8ac93cf9acd9898258349fc1720f60f8a', '--run-id', '1582000148684507', '--task-id', '1', '--input-paths', '1582000148684507/_parameters/0', '--cpu', '2', '--gpu', '0', '--memory', '2000', '--image', 'python:3.7', '--queue', 'arn:aws:batch:us-east-1:123456789535:job-queue/Metaflow-Job-Q', '--iam_role', 'arn:aws:iam::123456789535:role/Metaflow_Batch_ECS_Role_TEST', '--run-time-limit', '432000']

    The above are args it uses to call the step in a subprocess which will package the Job for batch for execution . Once on batch it runs something like this to invoke the processsing :

    'metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582000148684507 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}'

    Unit test for step should ideally work with persistent data in the datastore. The other thing I am assuming, is that you have set conda as environment manager. If conda is set, then you every subprocess which is being run, is having the same env as the other subprocesses so ideally any test run locally should work if packaged correctly for remote.

    I hope this makes sense. I am answering based on intuition and my understanding of metaflow. Please do let me know if you think I am incorrect somewhere.

    Jan Škoda
    @leftys

    @valayDave Unit tests should not run on some real data, but on artificially created data or several sets of data depending on the test case. So I need to mock the data, ideally together with the testing code for good readability/maintanability.

    I see some way to perform unit-testing: I would just run the single step method separately in the same process and mock input and output.

    For integration testing, the best I could think of is to run the flow externally by running a command, use data from datastore (thats acceptable for me for integration testing) and check results from local metadata store.

    Valay Dave
    @valayDave

    @leftys : "So I need to mock the data, ideally together with the testing code for good readability/maintanability." Exactly this. But In a linear flow, each step is will "Create" data for the next step. Now in such a situation, I am assuming that you will have to mock the data created by the flow at each step to unit test the step?

    Are you planning to use pytest for unit testing? If you are can u share some specs on how u will test individual steps for the flow? I also wanted to know if you have any strategy regarding autogenerating the mock data for each step

    Savin
    @savingoyal
    @valayDave How are you initializing the datastore? --datastore_root points to the actual location with all the needed artifacts for the user code to execute.
    @leftys @valayDave Usually folks unit test their business logic outside the scope of metaflow. Metaflow by itself has a fairly comprehensive suite of integration tests - https://github.com/Netflix/metaflow/tree/master/test
    Valay Dave
    @valayDave

    @savingoyal : What do u mean by "How are you initializing the datastore?". I have my current metaflow provider configured by metaflow configure aws. I have set S3 as the datastore and everything works fine with Batch. I am trying to understand the importance of the --data-root command-line argument.

    I was running my container-based isolated step without the environment variables of METAFLOW_DATASTORE_SYSROOT_S3. I was assumed that providing --datastore-root is the same as METAFLOW_DATASTORE_SYSROOT_S3 env var. If I ran with all env vars of that batch sets then the container-based isolated step works completely fine.

    Savin
    @savingoyal
    Ah I thought you ran into the error while writing your k8s plugin. Looks like a bug - we will investigate.
    Valay Dave
    @valayDave

    Yes, I actually did run into this error while developing a k8s plugin. I was trying to write a mock pod spec with which I would be able to run the job.

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: metaflow-runner
    spec:
      template:
        spec:
          containers:
            - name: testing-import
              image: python:3.7
              env:
                - name: AWS_ACCESS_KEY_ID
                  value: AAAAAAAAAAAAAAAAAAAA
                - name: AWS_SECRET_ACCESS_KEY
                  value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                - name: AWS_SESSION_TOKEN
                  value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                - name: METAFLOW_CODE_SHA
                  value: db02ca64a7b31d0abcefbd36b7858947989a7ac4
                - name: METAFLOW_CODE_URL
                  value: s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4
                - name: METAFLOW_CODE_DS
                  value: s3
                - name: METAFLOW_USER
                  value: valaydave
                - name: METAFLOW_SERVICE_URL
                  value: None
                - name: METAFLOW_SERVICE_HEADERS
                  value: "{}"
                - name: METAFLOW_DATASTORE_SYSROOT_LOCAL
                  value: ".metaflow"
                - name: METAFLOW_DATATOOLS_S3ROOT
                  value: s3://kubernetes-first-test-store/metaflow_store/data
                - name: METAFLOW_DEFAULT_DATASTORE
                  value: s3
                - name: METAFLOW_DEFAULT_METADATA
                  value: local
                - name: METAFLOW_INPUT_PATHS_0
                  value: 1582048815350055/_parameters/0
    
              command: ["sh"]
              args: ["-c",
              "set -e && echo 'Hello This is a Hi From Custom Library' && echo 'Setting up task environment.' && python -m pip install awscli click requests boto3                     --user -qqq && mkdir metaflow && cd metaflow && i=0; while [ $i -le 5 ]; do echo 'Downloading code package.'; python -m awscli s3 cp s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4 job.tar >/dev/null &&                         echo 'Code package downloaded.' && break; sleep 10; i=$((i+1));done  && tar xf job.tar && echo 'Bootstrapping environment.' && python -m metaflow.plugins.conda.batch_bootstrap HelloAWSFlow metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83 && echo 'Environment bootstrapped.' && echo 'Task is starting.' && metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582048815350055 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}"
              ]
          restartPolicy: Never
      backoffLimit: 4

    The above is a POD template that doesn't work because it doesn't have the METAFLOW_DATASTORE_SYSROOT_S3 env var.
    It give me this error :

    Hello This is a Hi From Custom Library
    Setting up task environment.
    Downloading code package.
    Code package downloaded.
    Bootstrapping environment.
    Environment bootstrapped.
    Task is starting.
        Data store error:
        Datastore root not found. Specify with DATASTORE_SYSROOT_S3 environment variable.

    But putting the METAFLOW_DATASTORE_SYSROOT_S3 env var, I works fine:

    Hello This is a Hi From Custom LibrarySetting up task environment.Downloading code package.Code package downloaded.
    Bootstrapping environment.
    Environment bootstrapped.
    Task is starting.
    Hellow From AWS

    I just wanted to know that do I need to always specify the env var even after putting command line args ?

    Savin
    @savingoyal
    Valay Dave
    @valayDave
    @savingoyal Thank you!. I will test it out with the patch.
    Ferdinand von den Eichen
    @fvde
    Hey guys short question, as I cannot for the life of me find it in the docs right now - what is the cli command to import a config file json?
    (EJ) Vivek Pandey
    @Viveckh
    Hey fellas! I had a quick question for folks adopting Veniqa for fairly decent sized projects. How are you ensuring modularity while also leveraging what metaflow has to offer? For example - I get sure we can import and use functions that perform business logic living in other non-Flow python files. But that would mean the variable states within those functions wouldn't be stored in metaflow's datastore.
    I am working on getting my team at a Fortune 100 Healthcare client to adopt Metaflow for the Data Science pipelines of a patent-pending platform.
    (EJ) Vivek Pandey
    @Viveckh
    @fvde Do you mean metaflow configure aws? But it doesn't import config, it only sets it. Importing should happen by default when you fire up a Flow as long as the config file is in the proper location i.e. ~/.metaflowconfig/config.json
    @Viveckh My bad on the typo - I meant to type Metaflow not Veniqa in my above msg
    Savin
    @savingoyal
    @Viveckh Within your flow, you can assign those variables to self. Not sure I totally follow your ask.
    (EJ) Vivek Pandey
    @Viveckh

    So let's say there is func1 in a separate business_logic.py file that is a regular python script and not a Flow

    In order to use func1 in my flow, can I just import it at the top of the flow globally or in a step as import func1 from business_logic and start calling it?

    seeravikiran
    @seeravikiran
    @Viveckh yes you can. to ensure that metaflow packages your code correctly, place your func1 file in the same directory as your flow code. I would probably place the import closest to where you might access the business logic (i.e. step)
    (EJ) Vivek Pandey
    @Viveckh
    It's probably not possible, but is there a way to make the states of the variables within that external function to be stored in the datastore.
    Valay Dave
    @valayDave

    Hello, I was trying to run Metaflow on Batch without the conda decorator and just use Dockerhub images of TensorFlow to run my DL Flow. But the start step of the flow crashed with this error :

    2020-02-20 19:59:43.011 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from .sidecar import SidecarSubProcess
    2020-02-20 19:59:43.012 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/sidecar.py", line 14, in <module>
    2020-02-20 19:59:43.012 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from .debug import debug
    2020-02-20 19:59:43.012 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/debug.py", line 7, in <module>
    2020-02-20 19:59:43.012 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     import metaflow.metaflow_config as config
    2020-02-20 19:59:43.012 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/metaflow/__init__.py", line 63, in <module>
    2020-02-20 19:59:43.013 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     _import_plugin_decorators(globals())
    2020-02-20 19:59:43.013 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/metaflow/decorators.py", line 398, in _import_plugin_decorators
    2020-02-20 19:59:43.013 [158225
    2020-02-20 19:59:46.620 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from metaflow.plugins.timeout_decorator import get_run_time_limit_for_task
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/plugins/__init__.py", line 21, in <module>
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from .conda.conda_step_decorator import CondaStepDecorator
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/plugins/conda/conda_step_decorator.py", line 21, in <module>
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from metaflow.datatools import S3
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/datatools/__init__.py", line 2, in <module>
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from .s3 import S3
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]   File "/metaflow/metaflow/datatools/s3.py", line 10, in <module>
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b]     from .. import current, FlowSpec
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] [693f9c92-4877-453b-b436-cfb56ba8a30b] ImportError: cannot import name FlowSpec
    2020-02-20 19:59:46.621 [1582253963241341/start/1 (pid 60508)] Batch error:
    2020-02-20 19:59:46.790 [1582253963241341/start/1 (pid 60508)] Task crashed. This could be a transient error. Use @retry to retry.

    The stack trace was too large to be printed here. Is it necessary to use conda with batch ?