Metaflow/community

Yes, I actually did run into this error while developing a k8s plugin. I was trying to write a mock pod spec with which I would be able to run the job.

apiVersion: batch/v1
kind: Job
metadata:
  name: metaflow-runner
spec:
  template:
    spec:
      containers:
        - name: testing-import
          image: python:3.7
          env:
            - name: AWS_ACCESS_KEY_ID
              value: AAAAAAAAAAAAAAAAAAAA
            - name: AWS_SECRET_ACCESS_KEY
              value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
            - name: AWS_SESSION_TOKEN
              value: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
            - name: METAFLOW_CODE_SHA
              value: db02ca64a7b31d0abcefbd36b7858947989a7ac4
            - name: METAFLOW_CODE_URL
              value: s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4
            - name: METAFLOW_CODE_DS
              value: s3
            - name: METAFLOW_USER
              value: valaydave
            - name: METAFLOW_SERVICE_URL
              value: None
            - name: METAFLOW_SERVICE_HEADERS
              value: "{}"
            - name: METAFLOW_DATASTORE_SYSROOT_LOCAL
              value: ".metaflow"
            - name: METAFLOW_DATATOOLS_S3ROOT
              value: s3://kubernetes-first-test-store/metaflow_store/data
            - name: METAFLOW_DEFAULT_DATASTORE
              value: s3
            - name: METAFLOW_DEFAULT_METADATA
              value: local
            - name: METAFLOW_INPUT_PATHS_0
              value: 1582048815350055/_parameters/0

          command: ["sh"]
          args: ["-c",
          "set -e && echo 'Hello This is a Hi From Custom Library' && echo 'Setting up task environment.' && python -m pip install awscli click requests boto3                     --user -qqq && mkdir metaflow && cd metaflow && i=0; while [ $i -le 5 ]; do echo 'Downloading code package.'; python -m awscli s3 cp s3://kubernetes-first-test-store/metaflow_store/HelloAWSFlow/data/db/db02ca64a7b31d0abcefbd36b7858947989a7ac4 job.tar >/dev/null &&                         echo 'Code package downloaded.' && break; sleep 10; i=$((i+1));done  && tar xf job.tar && echo 'Bootstrapping environment.' && python -m metaflow.plugins.conda.batch_bootstrap HelloAWSFlow metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83 && echo 'Environment bootstrapped.' && echo 'Task is starting.' && metaflow_HelloAWSFlow_linux-64_42c548a8c1def9ca0d877bf671025b5e69b97f83/bin/python -s -u hello.py --quiet --metadata local --environment conda --datastore s3 --event-logger nullSidecarLogger --monitor nullSidecarMonitor --datastore-root s3://kubernetes-first-test-store/metaflow_store --package-suffixes .py --pylint step start --run-id 1582048815350055 --task-id 1 --input-paths ${METAFLOW_INPUT_PATHS_0}"
          ]
      restartPolicy: Never
  backoffLimit: 4

The above is a POD template that doesn't work because it doesn't have the METAFLOW_DATASTORE_SYSROOT_S3 env var.
It give me this error :

Hello This is a Hi From Custom Library
Setting up task environment.
Downloading code package.
Code package downloaded.
Bootstrapping environment.
Environment bootstrapped.
Task is starting.
    Data store error:
    Datastore root not found. Specify with DATASTORE_SYSROOT_S3 environment variable.

But putting the METAFLOW_DATASTORE_SYSROOT_S3 env var, I works fine:

Hello This is a Hi From Custom LibrarySetting up task environment.Downloading code package.Code package downloaded.
Bootstrapping environment.
Environment bootstrapped.
Task is starting.
Hellow From AWS

I just wanted to know that do I need to always specify the env var even after putting command line args ?