Hydrospheredata/hydro-serving

maikefischer

@maikefischer

Where can I specify a proxy?

Bulat

@KineticCookie

Hello @maikefischer
If you have configured docker proxy at ~/.docker/config.json you can just mount this file to /root/.docker/config.json inside the manager container.

Otherwise you can set it up using this docker documentation -> https://docs.docker.com/network/proxy/ and mount the configuration file.

maikefischer

@maikefischer

@KineticCookie okay thanks! I configured my proxy in the ~/.docker/config.json file but I don't know how to mount it to the /root/.docker/config.json file inside the manager container. I did not find documentation on this.

Bulat

@KineticCookie

@maikefischer sorry for the late reply - didn't get the notification :(

it' depends on your cluster deployment method. e.g. you can edit your docker-compose file to mount the file. With kubernetes you can use ConfigMap mount in deployment.

https://docs.docker.com/compose/compose-file/#volumes
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/

mattmbk

@mattmbk

Hi @AgrawalAmey.
Sorry for the late reply. Monitoring service is not available as OSS right now, it's an enterprise feature. If you want, we can make a demo for you showing off monitoring service with various metrics, data profiling, reqstore. Please, submit a request via site https://hydrosphere.io/ for a more detailed discussion.

can i suggest updating https://hydrosphere.io/serving/ ? It says "Automated versioning, monitoring and testing functionality are included." and "Everything is open source. No premium features and enterprise versions. No hidden costs and learning curve can be as smooth as it gets."

Ilnur Garifullin

@tidylobster

Thank you for noticing that @mattmbk
We will update that soon.

Daniil Gannota

@danikgan

Hello everyone!

I am trying to finish this tutorial: https://hydrosphere.io/serving-docs/latest/getting-started/serving-simple-model.html, but I have the following error:

The models appear on localhost but cannot be used in the application:

Ilnur Garifullin

@tidylobster

Hi @danikgan
Seems like the runtime with tag 2.2.1 hasn't been published. Can you for now downgrade to hydrosphere/serving-runtime-python-3.7:2.1.0 in your serving.yaml?

Daniil Gannota

@danikgan

Thanks, that worked. I now have another issue:

Daniil Gannota

@danikgan

May I know how do tags affect the process assuming I only downloaded "hs v2.2.1"? Or shall I downgrade my hs and reinstall it?

@tidylobster

Ilnur Garifullin

@tidylobster

Tags shouldn't affect very much. Runtimes are pretty stable right now, and we just update them to keep in sync with the main version of Hydrosphere.

@KineticCookie do you have any ideas, why @danikgan is receiving 504?

Bulat

@KineticCookie

Hey @danikgan !
Could you please elaborate when do you encounter 504? What http endpoint returns the error?

Daniil Gannota

@danikgan

@KineticCookie sorry for late reply, didn't get a notification of your message.

Frankly, I am not entirely sure what's causing the error. I am running 'sudo hs -v upload'. Usually, I am on a bank's VPN but I discounted from them. Not sure what's causing the issue.

disconnected*

I even reinstall hs for the sack of reinstalling -- didn't help

although when I run hs--version I get 2.2.0 not 2.2.1

Daniil Gannota

@danikgan

http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 504 173

Daniil Gannota

@danikgan

PS Build logs don't show up anything as well

Bulat

@KineticCookie

sorry for late reply, didn't get a notification of your message.

Yeah, sometimes gitter does that. I didn't get them either ¯_(ツ)_/¯

There is possibility that the infrastructure didn't start up properly, and nginx can't pass requests.

Assuming that you deployed hydroserving on docker-compose, could you please send the result of docker ps -a with hydroserving containers and logs of postgres manager and managerui containers

@danikgan ping just in case gitter sends notification to the void

Daniil Gannota

@danikgan

@KineticCookie in case you like screenshots more:

Daniil Gannota

@danikgan

f9317f35aed2 f5895cade5d0 "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago angry_lederberg
82059c92a976 164367ad044d "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago objective_mcclintock
339e4f6ac31e 359e7316378e "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago musing_torvalds
d89f5281975e 3c997be69bd0 "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago upbeat_sinoussi
217b0f30723a 2241f577786f "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago keen_satoshi
d26cc60be162 hydrosphere/serving-manager-ui:2.2.0 "/bin/sh -c 'envsubs…" 3 weeks ago Up 45 seconds 0.0.0.0:80->80/tcp managerui
c450de23219e hydrosphere/serving-gateway:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds gateway
5088293cf1f4 hydrosphere/serving-manager:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds 0.0.0.0:19090->9090/tcp, 0.0.0.0:19091->9091/tcp manager
ef61cd999b3e postgres:9.6-alpine "docker-entrypoint.s…" 3 weeks ago Up 46 seconds postgres

This is the output after a restart of my laptop and an opening of docker and minikube.

Daniil Gannota

@danikgan

@KineticCookie @tidylobster hi there!

Bulat

@KineticCookie

Hi @danikgan
Sorry for the delay. Didn't get the notification 😕

Could you also send the result of docker logs d26cc60be162 and docker logs 5088293cf1f4

Daniil Gannota

@danikgan

I get a huge output because I ran Hydrosphere too many time, so let me try to send it as a text file

I don't think you'll need all of it, but I am sending anyways just in case

docker logs d26cc60be162.txt

docker logs 5088293cf1f4.txt

Daniil Gannota

@danikgan

@KineticCookie hope this notification finds you!;)

Daniil Gannota

@danikgan

SN Whenever I try to upload a model, it throws me either 504 gateway error or "model build failed" error after line "http://localhost:80 "GET /api/v2/model/version/linear_regression/24 HTTP/1.1" 200 717". I don't really know when it throws 504 and when build failure -- seems unpredictable to me.

Bulat

@KineticCookie

@danikgan hi!

I have just checked your logs

Thread starvation or clock leap detected.

Seems like there are not enough resources for containers and it keeps failing because of connection leaks

Could you try to allocate more resources to the containers and check if it fixes the issue?

Daniil Gannota

@danikgan

What would be the minimal requirements to run hs?

@KineticCookie

Daniil Gannota

@danikgan

I allocated 8GB to Docker and 4 cpus.
I allocated the same for minikube which I use for K8S.

I'll send you the latest logs now, but do you still think it's to do with hs running out of resources?

docker logs d26cc60be162.txt

docker logs 5088293cf1f4.txt

Daniil Gannota

@danikgan

debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/src
debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/increment_model
debug: Can't extract DVC metadata: No module named 'dvc'
Parsed model definition
Name: increment_model
Runtime: hydrosphere/serving-runtime-python-3.6:2.2.1
Install command: pip install -r requirements.txt
Host selector: None
Training data: None
Signature name: increment
Inputs:
| name   | shape   | type     | profile   |
|--------|---------|----------|-----------|
| number | [-1]    | DT_INT32 | NUMERICAL |
Outputs:
| name   | shape   | type     | profile   |
|--------|---------|----------|-----------|
| number | [-1]    | DT_INT32 | NUMERICAL |
Monitoring:
    None
Metadata:
Payload:
    /Users/danikgan/Desktop/Hydrosphere/increment_model/src
    /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
================================
debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/src as src
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt as requirements.txt
Uploading model to http://localhost
================================
debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "increment", "inputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}], "outputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.1", "name": "hydrosphere/serving-runtime-python-3.6"}, "name": "increment_model", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz'>)}
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 650
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/73/logs HTTP/1.1" 200 None
debug: Initialized SSE client from event source <Response [200]>
Build logs:
debug: Dispatching Log event #0, no data...

debug: Dispatching Log event #1, 60 bytes...
Step 1/6 : FROM hydrosphere/serving-runtime-python-3.6:2.2.1
debug: Dispatching Log event #2, 103 bytes...
manifest for hydrosphere/serving-runtime-python-3.6:2.2.1 not found: manifest unknown: manifest unknown
debug: Dispatching EndOfStream event, no data...

================================
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/increment_model/7 HTTP/1.1" 200 702
error: Model build failed

debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
debug: Can't extract DVC metadata: No module named 'dvc'
Parsed model definition
Name: linear_regression
Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
Install command: pip install -r requirements.txt
Host selector: None
Training data: None
Signature name: infer
Inputs:
| name   | shape   | type      | profile   |
|--------|---------|-----------|-----------|
| x      | [-1, 2] | DT_DOUBLE | NUMERICAL |
Outputs:
| name   | shape   | type      | profile   |
|--------|---------|-----------|-----------|
| y      | [-1]    | DT_DOUBLE | NUMERICAL |
Monitoring:
    None
Metadata:
Payload:
    /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
    /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
    /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
================================
debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
Uploading model to http://localhost
================================
debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 665
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/74/logs HTTP/1.1" 200 None
debug: Initialized SSE client from event source <Response [200]>
Build logs:
debug: Dispatching Log event #0, no data...

debug: Dispatching Log event #1, 60 bytes...
Step 1/6 : FROM hydrosphere/serving-runtime-python-3.7:2.2.0
debug: Dispatching Log event #2, 103 bytes...
manifest for hydrosphere/serving-runtime-python-3.7:2.2.0 not found: manifest unknown: manifest unknown
debug: Dispatching EndOfStream event, no data...

================================
debug: Starting new HTTP connection (1): localhost:80
debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/35 HTTP/1.1" 200 717
error: Model build failed

Even trying to upload different ML models produces exactly the same procedure for error

Bulat

@KineticCookie

Hi!
Seems like a problem here is that docker can't find the image.
I re-released the images, could you try again?

Daniil Gannota

@danikgan

Okay, so I deleted the old hs images, installed the new ones but still the same error

Could it be because I am on Helm 3? I need to find how to downgrade to v2.9 then

Where communities thrive

People

Repo info

Activity