Yeah, like the hyperlink to the cortex example of this model (bert). Here’s an example:
https://github.com/cortexlabs/cortex/tree/0.13/examples/tensorflow/sentiment-analyzer
https://github.com/cortexlabs/cortex/tree/0.13/examples/tensorflow/sentiment-analyzer
Oh sorry, I don't have this example in a git repo yet. I can work on doing that now. In the meantime, it is similar to the bert example in the repo you sent, except that I need to install bert-for-tf2. Here is the full error trace:
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2020-01-28 02:14:54.225961:cortex:INFO:loading the predictor from integrated_python_predictor.py
2020-01-28 02:14:55.374009:cortex:ERROR:failed to start api
Traceback (most recent call last):
File "/src/cortex/lib/context.py", line 124, in load_module
impl = imp.load_source(full_module_name, impl_path)
File "/usr/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/mnt/project/integrated_python_predictor.py", line 10, in <module>
from model_inference import inference_pipeline
File "/mnt/project/model_inference.py", line 4, in <module>
import bert
ModuleNotFoundError: No module named 'bert'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/src/cortex/python_serve/api.py", line 127, in start
predictor_class = ctx.get_predictor_class(api["name"], args.project_dir)
File "/src/cortex/lib/context.py", line 145, in get_predictor_class
"predictor", api["name"], os.path.join(project_dir, api["predictor"]["path"])
File "/src/cortex/lib/context.py", line 126, in load_module
raise UserException(str(e)) from e
cortex.lib.exceptions.UserException: error: api python-token-classifier: error in integrated_python_predictor.py: No module named 'bert'
Requirement already satisfied: tensorflow==2 in /usr/local/lib/python3.6/dist-packages (from -r /mnt/project/requirements.txt (line 1)) (2.0.0)
Collecting bert-for-tf2==0.13.4
Downloading https://files.pythonhosted.org/packages/75/15/03314e558f4c34642a90144fc8ec8bdbb0ef3c2ca80345007f9c7b007a07/bert-for-tf2-0.13.4.tar.gz (40kB)
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-xbgo5nb1/bert-for-tf2/pip-egg-info
cwd: /tmp/pip-install-xbgo5nb1/bert-for-tf2/
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-xbgo5nb1/bert-for-tf2/setup.py", line 21, in <module>
long_description = fh.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
----------------------------------------
Hmm, I didn’t get this with the same Python version locally. Out of curiosity, can you try with version
0.13.2 (or any older versions anyway)? I’ve also had this happen to me a while ago and I used this to solve mine: https://stackoverflow.com/a/21129492/2096747
I just tried 0.13.2 and got the same result
Although, it doesn’t answer your question of why it doesn’t work in your environment.
since that stack trace appears before the one that mentions
ascii, maybe this is the root cause
/mnt/project/model_inference.py should match model_inference.py in your local directory at the time you ran cortex deploy
Sorry, should have went to the beginning of the logs:
fetching logs...
2020-01-28 02:55:16.710501:cortex:INFO:downloading the project code
2020-01-28 02:55:16.796239:cortex:INFO:pulling the python serving image
Collecting git+git://github.com/kpe/bert-for-tf2.git@v0.13.4 (from -r /mnt/project/requirements.txt (line 2))
Cloning git://github.com/kpe/bert-for-tf2.git (to revision v0.13.4) to /tmp/pip-req-build-nsg4ej94
Running command git clone -q git://github.com/kpe/bert-for-tf2.git /tmp/pip-req-build-nsg4ej94
Running command git checkout -q 7acca1d6a4338cb24c4bc01f18caf52ef09d4298
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nsg4ej94/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nsg4ej94/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-nsg4ej94/pip-egg-info
cwd: /tmp/pip-req-build-nsg4ej94/
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-nsg4ej94/setup.py", line 21, in <module>
long_description = fh.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2020-01-28 02:55:20.203994:cortex:INFO:loading the predictor from integrated_python_predictor.py
2020-01-28 02:55:21.345960:cortex:ERROR:failed to start api
Traceback (most recent call last):
File "/src/cortex/lib/context.py", line 124, in load_module
impl = imp.load_source(full_module_name, impl_path)
File "/usr/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/mnt/project/integrated_python_predictor.py", line 10, in <module>
from model_inference import inference_pipeline
File "/mnt/project/model_inference.py", line 4, in <module>
import bert
ModuleNotFoundError: No module named 'bert'
Unfortunately, I am unsure if I can show you my source code right now, I will need to check with my boss to make sure. Sorry for the inconvenience. If it helps, I get the same unicode error when I add bert-for-tf2 to the requirements file of the iris-classifier tutorial
therefore it shouldn't be necessary to share your code, will keep you posted
I haven't found the issue yet, but wanted to give an update on my progress.
I am able to reproduce the error by running:
docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve
root@8d4af817edb2:/# pip install bert-for-tf2
<stack trace>However when I create a new colab notebook and update tensorflow via:
!pip install tensorflow==2.0.0(the version can be confirmed with import tensorflow; print(tensorflow.__version__))
And then I try !pip install bert-for-tf2 in the notebook, it succeeds. So there must be some dependency missing or with an incompatible version in the cortex docker image.
Please let me know if you come across a solution. Thank you again for your help!
i.e. it doesn't like something in the readme
I still don't know yet why it would error in the cortex docker container and not locally
$ docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve
root@45649019a21d:/# git clone https://github.com/kpe/bert-for-tf2.git
root@45649019a21d:/# cd bert-for-tf2
root@45649019a21d:/bert-for-tf2# git checkout v0.13.4
root@45649019a21d:/bert-for-tf2# python3
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("README.rst", "r") as fh:
... fh.read()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 9351: ordinal not in range(128)
reading a different file, like
LICENSE, succeeds
but on other systems (e.g. colab) either that is not a problem, or
fh.read() uses a difference decoder
docker run --rm -it --entrypoint=/bin/bash cortexlabs/python-serve:0.13.0
@ntorba I have created a PR for the fix (cortexlabs/cortex#784), which will be included in our next release. Thanks for bringing this to our attention!
Until we create the next release, you can use the updated docker image by adding this to your cluster.yaml file which you used when creating your cluster (or create a new one if you don't have one yet):
# cluster.yaml
# <other config here...>
image_python_serve: cortexlabsdev/python-serve:0.13.1-beta1
image_python_serve_gpu: cortexlabsdev/python-serve-gpu:0.13.1-beta1
image_onnx_serve: cortexlabsdev/onnx-serve:0.13.1-beta1
image_onnx_serve_gpu: cortexlabsdev/onnx-serve-gpu:0.13.1-beta1
image_tf_api: cortexlabsdev/tf-api:0.13.1-beta1You can then run cortex cluster update --config=cluster.yaml
Once the next version is released and you upgrade your cluster, you will be able to remove these fields.
Let us know if this works for you.
Also, thank you @RobertLucia for helping to debug!
Question: Apparently each time a new API is deployed (using cortex deploy), a new python image is deployed? (log says: pulling the python serving image). What about cluster update applied after launching the cluster up? (using cortex cluster update --config=cluster.yaml where I installed additional libraries). Are the changes still intact? I seems weird that a new python image is used each time an API is deployed. Thanks
@deliahu @RobertLucian The intstall of bert-for-tf2 now works with the new config! However, I am now running into another error:
... Above this in logs is more package installing
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.1.0,>=2.0.0->tensorflow==2.0.0->-r /mnt/project/requirements.txt (line 1)) (0.4.8)
Building wheels for collected packages: bert-for-tf2, py-params, params-flow
Building wheel for bert-for-tf2 (setup.py): started
Building wheel for bert-for-tf2 (setup.py): finished with status 'done'
Created wheel for bert-for-tf2: filename=bert_for_tf2-0.13.4-py3-none-any.whl size=29943 sha256=3d88ef9056a7567b31bdd9a0a301bc5a4090c43ea162a047c0a584022caa471d
Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/61/70/f4/6d4ad621bde1ba824e8f7f029c9110877e49c18eb10a5525d4
Building wheel for py-params (setup.py): started
Building wheel for py-params (setup.py): finished with status 'done'
Created wheel for py-params: filename=py_params-0.8.2-py3-none-any.whl size=4635 sha256=3d311bb1479b9fa980db76b3d9c966592ff235e3631a546687450fb733d263fd
Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/0c/dd/7b/849f2cccbb4e55cfc049615c12021182a39affa765fcf9e20b
Building wheel for params-flow (setup.py): started
Building wheel for params-flow (setup.py): finished with status 'done'
Created wheel for params-flow: filename=params_flow-0.7.4-py3-none-any.whl size=16194 sha256=a7df6356511e6c2aa6e09c043d2992fe093a1c09df709ed0900980c84c750754
Stored in directory: /tmp/pip-ephem-wheel-cache-r60345wa/wheels/f4/90/ca/a5d500c088762773489e532acd0268226cf2597f211928f38a
Successfully built bert-for-tf2 py-params params-flow
Installing collected packages: py-params, params-flow, bert-for-tf2
Successfully installed bert-for-tf2-0.13.4 params-flow-0.7.4 py-params-0.8.2
usage: serve.py [-h] --port PORT [--tf-serve-port TF_SERVE_PORT] --spec SPEC
[--model-dir MODEL_DIR] --cache-dir CACHE_DIR --project-dir
PROJECT_DIR
serve.py: error: the following arguments are required: --spec
@majeek every time an API replica is spun up, the image you are using for your API is downloaded from the container registry. I am not entirely sure what you mean by
What about cluster update applied after launching the cluster up? ... Are the changes still intact?. Let's say you are attempting to add system packages to your custom image and pushed to your image <registry_url>:latest. If your image_python_serve in cluster.yaml is <registry_url>:latest then you can cortex refresh <api_name> to use your new image without having to cortex cluster update --config=cluster.yaml because each replica always downloads the image from your container registry. You will have to do a cortex cluster update --config=cluster.yaml if you are changing the image or tag. For example, instead of building your images and pushing to the same tag (e.g. latest), you push to a different version every time then you need to do a cortex cluster update --config=cluster.yaml. Let me know if this addresses your question, otherwise please feel free to rephrase your question.
@majeek It seems like you are using Cortex v0.12. Cortex zips the entire working directory containing your cortex.yaml file and pushes the zipped directory to S3. On replica initialization, your project files are downloaded from S3 and unzipped. What kind of files are in your working directory that you want to be included in the zip? If there are large files that need to be in included in your API such as labels, we recommend pushing them to cloud storage like S3 and downloading them from S3.
If storing some of the files in your project directory in cloud storage and downloading them doesn't work for you. Then upgrading to Cortex v0.13 will also resolve this issue because the hard file size limit has been lifted in Cortex v0.13.
@ntorba The
--spec arg is supposed to be set by the Cortex Operator. What version of Cortex are you using? The --spec flag was introduced in Cortex v0.13. Older versions of Cortex Operator are incompatible with images based on Cortex v0.13 which is why you may be running into this issue.