@danikgan hi!
I have just checked your logs
Thread starvation or clock leap detected.
Seems like there are not enough resources for containers and it keeps failing because of connection leaks
Could you try to allocate more resources to the containers and check if it fixes the issue?
debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/src
debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/increment_model
debug: Can't extract DVC metadata: No module named 'dvc'
Parsed model definition
Name: increment_model
Runtime: hydrosphere/serving-runtime-python-3.6:2.2.1
Install command: pip install -r requirements.txt
Host selector: None
Training data: None
Signature name: increment
Inputs:
| name | shape | type | profile |
|--------|---------|----------|-----------|
| number | [-1] | DT_INT32 | NUMERICAL |
Outputs:
| name | shape | type | profile |
|--------|---------|----------|-----------|
| number | [-1] | DT_INT32 | NUMERICAL |
Monitoring:
None
Metadata:
Payload:
/Users/danikgan/Desktop/Hydrosphere/increment_model/src
/Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
================================
debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/src as src
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt as requirements.txt
Uploading model to http://localhost
================================
debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "increment", "inputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}], "outputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.1", "name": "hydrosphere/serving-runtime-python-3.6"}, "name": "increment_model", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz'>)}
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 650
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/73/logs HTTP/1.1" 200 None
debug: Initialized SSE client from event source <Response [200]>
Build logs:
debug: Dispatching Log event #0, no data...
debug: Dispatching Log event #1, 60 bytes...
Step 1/6 : FROM hydrosphere/serving-runtime-python-3.6:2.2.1
debug: Dispatching Log event #2, 103 bytes...
manifest for hydrosphere/serving-runtime-python-3.6:2.2.1 not found: manifest unknown: manifest unknown
debug: Dispatching EndOfStream event, no data...
================================
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/increment_model/7 HTTP/1.1" 200 702
error: Model build failed
debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
debug: Can't extract DVC metadata: No module named 'dvc'
Parsed model definition
Name: linear_regression
Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
Install command: pip install -r requirements.txt
Host selector: None
Training data: None
Signature name: infer
Inputs:
| name | shape | type | profile |
|--------|---------|-----------|-----------|
| x | [-1, 2] | DT_DOUBLE | NUMERICAL |
Outputs:
| name | shape | type | profile |
|--------|---------|-----------|-----------|
| y | [-1] | DT_DOUBLE | NUMERICAL |
Monitoring:
None
Metadata:
Payload:
/Users/danikgan/Desktop/Hydrosphere/linear_regression/src
/Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
/Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
================================
debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
Uploading model to http://localhost
================================
debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 665
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/74/logs HTTP/1.1" 200 None
debug: Initialized SSE client from event source <Response [200]>
Build logs:
debug: Dispatching Log event #0, no data...
debug: Dispatching Log event #1, 60 bytes...
Step 1/6 : FROM hydrosphere/serving-runtime-python-3.7:2.2.0
debug: Dispatching Log event #2, 103 bytes...
manifest for hydrosphere/serving-runtime-python-3.7:2.2.0 not found: manifest unknown: manifest unknown
debug: Dispatching EndOfStream event, no data...
================================
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/35 HTTP/1.1" 200 717
error: Model build failed
debug: Dispatching Log event #406, 10 bytes...
Extracting
debug: Dispatching Log event #407, 13 bytes...
Pull complete
debug: Dispatching Log event #408, 79 bytes...
Digest: sha256:e62e47b3352b23788e4e31f3e4a5636328a17cc65e7d44b5e234a6f3868cd977
debug: Dispatching Log event #409, 79 bytes...
Status: Downloaded newer image for hydrosphere/serving-runtime-python-3.7:2.2.0
debug: Dispatching Log event #410, 17 bytes...
---> 90f743fd43f9
debug: Dispatching Log event #411, 45 bytes...
Step 2/6 : LABEL MODEL_NAME=linear_regression
debug: Dispatching Log event #412, 28 bytes...
---> Running in 1373c319bfc5
debug: Dispatching Log event #413, 44 bytes...
Removing intermediate container 1373c319bfc5
debug: Dispatching Log event #414, 17 bytes...
---> dc43c044b0a8
debug: Dispatching Log event #415, 32 bytes...
Step 3/6 : LABEL MODEL_VERSION=2
debug: Dispatching Log event #416, 28 bytes...
---> Running in 34b07afa09bb
debug: Dispatching Log event #417, 44 bytes...
Removing intermediate container 34b07afa09bb
debug: Dispatching Log event #418, 17 bytes...
---> 8d29ff4fe4dc
debug: Dispatching Log event #419, 27 bytes...
Step 4/6 : ADD model /model
debug: Dispatching Log event #420, 17 bytes...
---> bdeb5bca1ddb
debug: Dispatching Log event #421, 31 bytes...
Step 5/6 : WORKDIR /model/files
debug: Dispatching Log event #422, 28 bytes...
---> Running in 9aaea0b13b3f
debug: Dispatching Log event #423, 44 bytes...
Removing intermediate container 9aaea0b13b3f
debug: Dispatching Log event #424, 17 bytes...
---> a86bfaf28563
debug: Dispatching Log event #425, 46 bytes...
Step 6/6 : RUN pip install -r requirements.txt
debug: Dispatching Log event #426, 28 bytes...
---> Running in 0aa6ee58f721
debug: Dispatching message event, no data...
debug: Dispatching message event, no data...
debug: Dispatching Log event #427, 332 bytes...
[91mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef890a50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
debug: Dispatching message event, no data...
debug: Dispatching message event, no data...
debug: Dispatching Log event #428, 332 bytes...
[91mWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7490>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
debug: Dispatching message event, no data...
debug: Dispatching message event, no data...
debug: Dispatching Log event #429, 332 bytes...
[91mWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7590>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
debug: Dispatching message event, no data...
debug: Dispatching message event, no data...
debug: Dispatching Log event #430, 332 bytes...
[91mWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
debug: Dispatching message event, no data...
debug: Dispatching message event, no data...
debug: Dispatching Log event #431, 332 bytes...
[91mWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7dd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
debug: Dispatching message event, no data.
hs -v upload
🤔
Seems like it can't install dependency from requirements.txt
Do you have some specific network settings, which could deny egress traffic from contrainers? Or you installed hydroserving from docker-compose.yaml
as-is?
People encountered similar issues, and seems like it's related to dns lookup on a host with docker daemon: https://forums.docker.com/t/facing-issue-with-creating-angular-application-image-using-docker/87270
Yes, I do indeed have a custom network setting. If it can't install the dependencies, might this mean an issue with pip?
And yes, I installed hydro serving from docker-compose.yaml
as is. I tried to configure it to use my pip setting: under docker-compose.yaml
I have this line:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./pip:/root/.pip
pip.conf
and certificate from my workplace.
docker-compose up
and connect to the shell of manager container, I find that hydrosphere does not have pip installed.
@danikgan You're on a right track, but you don't need to manipulate docker-compose.yaml
to deliver your pip.conf
file to the build context. One way of doing that might be just to include that file in the serving.yaml
file's payload and update install-comand (which is just a shell script). If we're talking about our linear_regression example, then it will look like this:
kind: Model
name: linear_regression
runtime: "hydrosphere/serving-runtime-python-3.6:2.2.1"
install-command: "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt"
payload:
- "src/"
- "requirements.txt"
- "pip.conf" # specify location of your pip.conf
- "model.h5"
contract:
name: infer
inputs:
x:
shape: [-1, 2]
type: double
profile: numerical
outputs:
y:
shape: [-1]
type: double
profile: numerical
In this case I've just specified that pip.conf
file lies within my current build context. So, pip should utilize pip.conf and you won't see Retrying (Retry(total=1, connect=None...
during hs -v upload
.
pip.conf
into payload of your model, you can update install-command pip install -r requirements.txt
to include all network settings required for your environment. For instance, pip install --proxy <proxy> -r requirements.txt
.
Alright, we're on the right track -- that's great! Here's what I tried doing:
serving.yaml
file as specified.pip.conf
file as cert = terra_bin.cert
I still get an error though (and I tried various combinations I could think of to fix this). Could it be an issue with accepting the certificate? The pip.conf
file should be correct as this is the same one I am using for my own local pip configuration (which works btw).
debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
debug: Payload pip.conf is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
debug: Payload tecca_bin.cer is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
debug: Can't extract DVC metadata: No module named 'dvc'
Parsed model definition
Name: linear_regression
Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
Install command: PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt
Host selector: None
Training data: None
Signature name: infer
Inputs:
| name | shape | type | profile |
|--------|---------|-----------|-----------|
| x | [-1, 2] | DT_DOUBLE | NUMERICAL |
Outputs:
| name | shape | type | profile |
|--------|---------|-----------|-----------|
| y | [-1] | DT_DOUBLE | NUMERICAL |
Monitoring:
None
Metadata:
Payload:
/Users/danikgan/Desktop/Hydrosphere/linear_regression/src
/Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
/Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
/Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
/Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
================================
debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf as pip.conf
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer as tecca_bin.cer
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
Uploading model to http://localhost
================================
debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/11/logs HTTP/1.1" 504 173
error: Got server error <Response [504]>
Error: Server returned an error: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>openresty/1.15.8.3</center>
</body>
</html>
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/12/logs HTTP/1.1" 404 74
warning: Build logs are not available
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/12 HTTP/1.1" 200 742
error: Model build failed
{"model": {"id": 1, "name": "linear_regression"}, "image": {"name": "linear_regression", "tag": "12"}, "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "finished": "2020-06-03T11:25:24.405Z", "modelContract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "x", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}}], "outputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "y", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}}]}}, "id": 12, "status": "Failed", "metadata": {}, "kind": "Internal", "modelVersion": 12, "runtime": {"name": "hydrosphere/serving-runtime-python-3.7", "tag": "2.2.1"}, "created": "2020-06-03T11:25:19.746Z"}
hydrosphere/serving-runtime-python-3.6
, respectively.
/model/files/pip.conf
, /model/files/cert.pem
paths. So, you might try to update your pip.conf
to use the correct path to your cert.pem.
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 504 173
error: Got server error <Response [504]>
Error: Server returned an error: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>openresty/1.15.8.3</center>
</body>
</html>
Hi, all. I have started the quickstart documentation for the hs cli on my Windows10 machine. I installed hs==2.4.0 and am getting this error
hs --version
Traceback (most recent call last):
File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\K27793\AppData\Roaming\Python\Python36\Scripts\hs.exe\__main__.py", line 5, in <module>
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\commands\__init__.py", line 1, in <module>
from hydroserving.cli.commands.hs import *
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\commands\hs.py", line 7, in <module>
from hydroserving.cli.context_object import ContextObject
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\context_object.py", line 7, in <module>
from hydroserving.core.apply import ApplyService
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\core\apply.py", line 8, in <module>
from hydroserving.core.deployment_config.parser import parse_deployment_configuration
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\core\deployment_config\parser.py", line 3, in <module>
from hydrosdk import DeploymentConfiguration
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\__init__.py", line 3, in <module>
from hydrosdk.modelversion import LocalModel, ModelVersion
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\modelversion.py", line 19, in <module>
from hydrosdk.contract import ModelContract_to_contract_dict, contract_dict_to_ModelContract, validate_contract
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\contract.py", line 11, in <module>
from hydrosdk.data.types import alias_to_proto_dtype, shape_to_proto, PY_TO_DTYPE, np_to_proto_dtype, proto_to_np_dtype
File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\data\types.py", line 130, in <module>
np.float128: None,
AttributeError: module 'numpy' has no attribute 'float128'
which is related to this issue winpython/winpython#613.
Seems like an issue with hydro-sdk on this line https://github.com/Hydrospheredata/hydro-serving-sdk/blob/master/hydrosdk/data/types.py#L130
Downgrading to hs==2.2.0, which doesn't have hydro-sdk dependency, worked on Windows10.
Hi @gaopan1996 , thanks for noticing this issue! We indeed do not perform multi-platform testing of our client libraries, so we are mostly unaware of possible issues for our Windows users.
Since this is an issue, I suggest you posting the same text you’ve sent us here to our Github issues via this link: https://github.com/Hydrospheredata/hydro-serving/issues/new
I'll return shortly after with a solution and post it directly to the issue you will have created.