Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    ntorba
    @ntorba
    Yes. Which file?
    Robert Lucian Chiriac
    @RobertLucian
    Can you show me the link to the example?
    ntorba
    @ntorba
    I'm sorry, not quite sure what you mean. A link?
    Robert Lucian Chiriac
    @RobertLucian
    Yeah, like the hyperlink to the cortex example of this model (bert). Here’s an example:
    https://github.com/cortexlabs/cortex/tree/0.13/examples/tensorflow/sentiment-analyzer
    ntorba
    @ntorba
    Oh sorry, I don't have this example in a git repo yet. I can work on doing that now. In the meantime, it is similar to the bert example in the repo you sent, except that I need to install bert-for-tf2. Here is the full error trace:
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    2020-01-28 02:14:54.225961:cortex:INFO:loading the predictor from integrated_python_predictor.py
    2020-01-28 02:14:55.374009:cortex:ERROR:failed to start api
    Traceback (most recent call last):
      File "/src/cortex/lib/context.py", line 124, in load_module
        impl = imp.load_source(full_module_name, impl_path)
      File "/usr/lib/python3.6/imp.py", line 172, in load_source
        module = _load(spec)
      File "<frozen importlib._bootstrap>", line 684, in _load
      File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/mnt/project/integrated_python_predictor.py", line 10, in <module>
        from model_inference import inference_pipeline
      File "/mnt/project/model_inference.py", line 4, in <module>
        import bert
    ModuleNotFoundError: No module named 'bert'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/src/cortex/python_serve/api.py", line 127, in start
        predictor_class = ctx.get_predictor_class(api["name"], args.project_dir)
      File "/src/cortex/lib/context.py", line 145, in get_predictor_class
        "predictor", api["name"], os.path.join(project_dir, api["predictor"]["path"])
      File "/src/cortex/lib/context.py", line 126, in load_module
        raise UserException(str(e)) from e
    cortex.lib.exceptions.UserException: error: api python-token-classifier: error in integrated_python_predictor.py: No module named 'bert'
    Requirement already satisfied: tensorflow==2 in /usr/local/lib/python3.6/dist-packages (from -r /mnt/project/requirements.txt (line 1)) (2.0.0)
    Collecting bert-for-tf2==0.13.4
      Downloading https://files.pythonhosted.org/packages/75/15/03314e558f4c34642a90144fc8ec8bdbb0ef3c2ca80345007f9c7b007a07/bert-for-tf2-0.13.4.tar.gz (40kB)
        ERROR: Command errored out with exit status 1:
         command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-xbgo5nb1/bert-for-tf2/pip-egg-info
             cwd: /tmp/pip-install-xbgo5nb1/bert-for-tf2/
        Complete output (7 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py", line 21, in <module>
            long_description = fh.read()
          File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
            return codecs.ascii_decode(input, self.errors)[0]
        UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
        ----------------------------------------
    Robert Lucian Chiriac
    @RobertLucian
    I just realized you’re trying to intall this:
    https://pypi.org/project/bert-for-tf2/
    ntorba
    @ntorba
    Yes, sorry, I should have specified that.
    Robert Lucian Chiriac
    @RobertLucian
    Give me a second, let me try installing this locally.
    ntorba
    @ntorba
    Thank you
    Robert Lucian Chiriac
    @RobertLucian
    Hmm, I didn’t get this with the same Python version locally. Out of curiosity, can you try with version 0.13.2 (or any older versions anyway)? I’ve also had this happen to me a while ago and I used this to solve mine: https://stackoverflow.com/a/21129492/2096747
    ntorba
    @ntorba
    I am confused, if I get this error when launching a deployment, which file of mine could be causing the unicode error?
    I just tried 0.13.2 and got the same result
    Robert Lucian Chiriac
    @RobertLucian
    You could try pip installing it from source, thus bypassing the archive from pypi. Maybe there’s something bad in it. Like this pip install -U git+git://github.com/kpe/bert-for-tf2.git@v0.13.4.
    Although, it doesn’t answer your question of why it doesn’t work in your environment.
    ntorba
    @ntorba
    So should I put that github link in my requirements.txt file?
    Robert Lucian Chiriac
    @RobertLucian
    Yeah, you’d just copy paste the git+git://github.com/kpe/bert-for-tf2.git@v0.13.4 inside the requirements.txt file.
    David Eliahu
    @deliahu
    Is it possible that this portion of your stack trace is the culprit?
      File "/mnt/project/model_inference.py", line 4, in <module>
        import bert
    ModuleNotFoundError: No module named 'bert'
    since that stack trace appears before the one that mentions ascii, maybe this is the root cause
    /mnt/project/model_inference.py should match model_inference.py in your local directory at the time you ran cortex deploy
    ntorba
    @ntorba
    Sorry, should have went to the beginning of the logs:
    fetching logs...
    2020-01-28 02:55:16.710501:cortex:INFO:downloading the project code
    2020-01-28 02:55:16.796239:cortex:INFO:pulling the python serving image
    Collecting git+git://github.com/kpe/bert-for-tf2.git@v0.13.4 (from -r /mnt/project/requirements.txt (line 2))
      Cloning git://github.com/kpe/bert-for-tf2.git (to revision v0.13.4) to /tmp/pip-req-build-nsg4ej94
      Running command git clone -q git://github.com/kpe/bert-for-tf2.git /tmp/pip-req-build-nsg4ej94
      Running command git checkout -q 7acca1d6a4338cb24c4bc01f18caf52ef09d4298
        ERROR: Command errored out with exit status 1:
         command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nsg4ej94/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nsg4ej94/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-nsg4ej94/pip-egg-info
             cwd: /tmp/pip-req-build-nsg4ej94/
        Complete output (7 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-req-build-nsg4ej94/setup.py", line 21, in <module>
            long_description = fh.read()
          File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
            return codecs.ascii_decode(input, self.errors)[0]
        UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
        ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    2020-01-28 02:55:20.203994:cortex:INFO:loading the predictor from integrated_python_predictor.py
    2020-01-28 02:55:21.345960:cortex:ERROR:failed to start api
    Traceback (most recent call last):
      File "/src/cortex/lib/context.py", line 124, in load_module
        impl = imp.load_source(full_module_name, impl_path)
      File "/usr/lib/python3.6/imp.py", line 172, in load_source
        module = _load(spec)
      File "<frozen importlib._bootstrap>", line 684, in _load
      File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/mnt/project/integrated_python_predictor.py", line 10, in <module>
        from model_inference import inference_pipeline
      File "/mnt/project/model_inference.py", line 4, in <module>
        import bert
    ModuleNotFoundError: No module named 'bert'
    Robert Lucian Chiriac
    @RobertLucian
    Could you show us your source code?
    David Eliahu
    @deliahu
    thanks for sharing, it seems like this is an issue with the pip install, do you mind sharing if you're running on a CPU instance (e.g. m5.large) or a gpu instance (e.g. g4dn.xlarge)?
    ntorba
    @ntorba
    I am running a m5.large cpu instance
    Unfortunately, I am unsure if I can show you my source code right now, I will need to check with my boss to make sure. Sorry for the inconvenience. If it helps, I get the same unicode error when I add bert-for-tf2 to the requirements file of the iris-classifier tutorial
    David Eliahu
    @deliahu
    yes that does help, that's what I am in the process of reproducing now :)
    therefore it shouldn't be necessary to share your code, will keep you posted
    ntorba
    @ntorba
    Thanks!
    David Eliahu
    @deliahu

    I haven't found the issue yet, but wanted to give an update on my progress.

    I am able to reproduce the error by running:

    docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve
    root@8d4af817edb2:/# pip install bert-for-tf2
    
    <stack trace>

    However when I create a new colab notebook and update tensorflow via:

    !pip install tensorflow==2.0.0

    (the version can be confirmed with import tensorflow; print(tensorflow.__version__))

    And then I try !pip install bert-for-tf2 in the notebook, it succeeds. So there must be some dependency missing or with an incompatible version in the cortex docker image.

    ntorba
    @ntorba
    Weird. I was able to successfully install bert-for-tf2 locally as well with tensorflow 2
    Please let me know if you come across a solution. Thank you again for your help!
    David Eliahu
    @deliahu
    i.e. it doesn't like something in the readme
    I still don't know yet why it would error in the cortex docker container and not locally
    David Eliahu
    @deliahu
    Another update: I am able to reproduce it now even more minimally, like this:
    $ docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve
    root@45649019a21d:/# git clone https://github.com/kpe/bert-for-tf2.git
    root@45649019a21d:/# cd bert-for-tf2
    root@45649019a21d:/bert-for-tf2# git checkout v0.13.4
    root@45649019a21d:/bert-for-tf2# python3
    Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
    [GCC 8.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> with open("README.rst", "r") as fh:
    ...   fh.read()
    ... 
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
    reading a different file, like LICENSE, succeeds
    David Eliahu
    @deliahu
    So it does seem to be an issue with the README containing non-ascii characters
    but on other systems (e.g. colab) either that is not a problem, or fh.read() uses a difference decoder
    David Eliahu
    @deliahu
    edit on the above set of commands (thanks @RobertLucian): the first command should be:
    docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve:0.13.0
    David Eliahu
    @deliahu

    @ntorba I have created a PR for the fix (cortexlabs/cortex#784), which will be included in our next release. Thanks for bringing this to our attention!

    Until we create the next release, you can use the updated docker image by adding this to your cluster.yaml file which you used when creating your cluster (or create a new one if you don't have one yet):

    # cluster.yaml
    
    # <other config here...>
    
    image_python_serve: cortexlabsdev/python-serve:0.13.1-beta1
    image_python_serve_gpu: cortexlabsdev/python-serve-gpu:0.13.1-beta1
    image_onnx_serve: cortexlabsdev/onnx-serve:0.13.1-beta1
    image_onnx_serve_gpu: cortexlabsdev/onnx-serve-gpu:0.13.1-beta1
    image_tf_api: cortexlabsdev/tf-api:0.13.1-beta1

    You can then run cortex cluster update --config=cluster.yaml

    Once the next version is released and you upgrade your cluster, you will be able to remove these fields.

    Let us know if this works for you.

    Also, thank you @RobertLucia for helping to debug!
    Majeed Kassis
    @majeek
    Question: Apparently each time a new API is deployed (using cortex deploy), a new python image is deployed? (log says: pulling the python serving image). What about cluster update applied after launching the cluster up? (using cortex cluster update --config=cluster.yaml where I installed additional libraries). Are the changes still intact? I seems weird that a new python image is used each time an API is deployed. Thanks
    ntorba
    @ntorba
    @deliahu @RobertLucian The intstall of bert-for-tf2 now works with the new config! However, I am now running into another error:
    ... Above this in logs is more package installing
    
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow==2.0.0->-r /mnt/project/requirements.txt (line 1)) (0.4.8)
    Building wheels for collected packages: bert-for-tf2, py-params, params-flow
      Building wheel for bert-for-tf2 (setup.py): started
      Building wheel for bert-for-tf2 (setup.py): finished with status 'done'
      Created wheel for bert-for-tf2: filename=bert_for_tf2-0.13.4-py3-none-any.whl size=29943 sha256=3d88ef9056a7567b31bdd9a0a301bc5a4090c43ea162a047c0a584022caa471d
      Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/61/70/f4/6d4ad621bde1ba824e8f7f029c9110877e49c18eb10a5525d4
      Building wheel for py-params (setup.py): started
      Building wheel for py-params (setup.py): finished with status 'done'
      Created wheel for py-params: filename=py_params-0.8.2-py3-none-any.whl size=4635 sha256=3d311bb1479b9fa980db76b3d9c966592ff235e3631a546687450fb733d263fd
      Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/0c/dd/7b/849f2cccbb4e55cfc049615c12021182a39affa765fcf9e20b
      Building wheel for params-flow (setup.py): started
      Building wheel for params-flow (setup.py): finished with status 'done'
      Created wheel for params-flow: filename=params_flow-0.7.4-py3-none-any.whl size=16194 sha256=a7df6356511e6c2aa6e09c043d2992fe093a1c09df709ed0900980c84c750754
      Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/f4/90/ca/a5d500c088762773489e532acd0268226cf2597f211928f38a
    Successfully built bert-for-tf2 py-params params-flow
    Installing collected packages: py-params, params-flow, bert-for-tf2
    Successfully installed bert-for-tf2-0.13.4 params-flow-0.7.4 py-params-0.8.2
    usage: serve.py [-h] --port PORT [--tf-serve-port TF_SERVE_PORT] --spec SPEC
                    [--model-dir MODEL_DIR] --cache-dir CACHE_DIR --project-dir
                    PROJECT_DIR
    serve.py: error: the following arguments are required: --spec
    ntorba
    @ntorba
    I am not sure if the --spec arg is something I am missing in in my config files
    Majeed Kassis
    @majeek
    Another question: error: zipped project folder exceeds 52428800 bytes <- how to overcome this limitation? Thanks
    Vishal Bollu
    @vishalbollu
    @majeek every time an API replica is spun up, the image you are using for your API is downloaded from the container registry. I am not entirely sure what you mean by What about cluster update applied after launching the cluster up? ... Are the changes still intact?. Let's say you are attempting to add system packages to your custom image and pushed to your image <registry_url>:latest. If your image_python_serve in cluster.yaml is <registry_url>:latest then you can cortex refresh <api_name> to use your new image without having to cortex cluster update --config=cluster.yaml because each replica always downloads the image from your container registry. You will have to do a cortex cluster update --config=cluster.yaml if you are changing the image or tag. For example, instead of building your images and pushing to the same tag (e.g. latest), you push to a different version every time then you need to do a cortex cluster update --config=cluster.yaml. Let me know if this addresses your question, otherwise please feel free to rephrase your question.
    Vishal Bollu
    @vishalbollu

    @majeek It seems like you are using Cortex v0.12. Cortex zips the entire working directory containing your cortex.yaml file and pushes the zipped directory to S3. On replica initialization, your project files are downloaded from S3 and unzipped. What kind of files are in your working directory that you want to be included in the zip? If there are large files that need to be in included in your API such as labels, we recommend pushing them to cloud storage like S3 and downloading them from S3.

    If storing some of the files in your project directory in cloud storage and downloading them doesn't work for you. Then upgrading to Cortex v0.13 will also resolve this issue because the hard file size limit has been lifted in Cortex v0.13.

    Vishal Bollu
    @vishalbollu
    @ntorba The --spec arg is supposed to be set by the Cortex Operator. What version of Cortex are you using? The --spec flag was introduced in Cortex v0.13. Older versions of Cortex Operator are incompatible with images based on Cortex v0.13 which is why you may be running into this issue.