Hydrospheredata/hydro-serving

Chaitra1996

@Chaitra1996

Hii, i have installed hydro serving on docker cluster with version 2.0.4. when am trying to upload model, am getting model build failed error. Could someone help me out with this issue.

Ilnur Garifullin

@tidylobster

Hi @Chaitra1996 , can you perform hs -v upload for more verbose logs?

Chaitra1996

@Chaitra1996

Hi @tidylobster i tried the command that u asked for. I am able to see the verbose logs. As shown in the image i tried to resolve the issue by setting the environment variable for git executable. But, still the error continues.

Ilnur Garifullin

@tidylobster

Well, hs just tries to parse .git and .dvc metadata, it's not essential for the model upload. Can you share full logs of the model upload? What is happening when the archive is packed and is being uploaded to the serving?

Bulat

@KineticCookie

Hi @Chaitra1996
This error occured because manager service encountered some issue while processing you model files. To dig deeper into it, we'll need to look for manager service logs - they should contain progress of model processing.

In addition to that, in the latest dev version we implemented logs passthrough, so they'll be available to you via CLI or web UI. We'll release it as soon as we stabilize all the services. Looking forward to notify you when it's done! :)

Chaitra1996

@Chaitra1996

@tidylobster the above two images are the logs that have appeared after running the upload command. And its not getting uploaded on to serving.

Chaitra1996

@Chaitra1996

Hi @KineticCookie , in the current version we will not be able to look for the logs?

Bulat

@KineticCookie

@Chaitra1996 you can look for the container/pod logs, but it would require using kubectl.
And yes, logs API (if I can call that an API) unfortunately is not in the 2.0.4 release. :(

Chaitra1996

@Chaitra1996

@KineticCookie Thank you

maikefischer

@maikefischer

Hi, I am following the Quickstart guide from https://hydrosphere.io/serving-docs/latest/tutorials/python.html. I am encountering a problem when I want to upload my model to the local serving cluster (hs upload). My build is failing due to this error:

Where can I specify a proxy?

Bulat

@KineticCookie

Hello @maikefischer
If you have configured docker proxy at ~/.docker/config.json you can just mount this file to /root/.docker/config.json inside the manager container.

Otherwise you can set it up using this docker documentation -> https://docs.docker.com/network/proxy/ and mount the configuration file.

maikefischer

@maikefischer

@KineticCookie okay thanks! I configured my proxy in the ~/.docker/config.json file but I don't know how to mount it to the /root/.docker/config.json file inside the manager container. I did not find documentation on this.

Bulat

@KineticCookie

@maikefischer sorry for the late reply - didn't get the notification :(

it' depends on your cluster deployment method. e.g. you can edit your docker-compose file to mount the file. With kubernetes you can use ConfigMap mount in deployment.

https://docs.docker.com/compose/compose-file/#volumes
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/

mattmbk

@mattmbk

Hi @AgrawalAmey.
Sorry for the late reply. Monitoring service is not available as OSS right now, it's an enterprise feature. If you want, we can make a demo for you showing off monitoring service with various metrics, data profiling, reqstore. Please, submit a request via site https://hydrosphere.io/ for a more detailed discussion.

can i suggest updating https://hydrosphere.io/serving/ ? It says "Automated versioning, monitoring and testing functionality are included." and "Everything is open source. No premium features and enterprise versions. No hidden costs and learning curve can be as smooth as it gets."

Ilnur Garifullin

@tidylobster

Thank you for noticing that @mattmbk
We will update that soon.

Daniil Gannota

@danikgan

Hello everyone!

I am trying to finish this tutorial: https://hydrosphere.io/serving-docs/latest/getting-started/serving-simple-model.html, but I have the following error:

The models appear on localhost but cannot be used in the application:

Ilnur Garifullin

@tidylobster

Hi @danikgan
Seems like the runtime with tag 2.2.1 hasn't been published. Can you for now downgrade to hydrosphere/serving-runtime-python-3.7:2.1.0 in your serving.yaml?

Daniil Gannota

@danikgan

Thanks, that worked. I now have another issue:

Daniil Gannota

@danikgan

May I know how do tags affect the process assuming I only downloaded "hs v2.2.1"? Or shall I downgrade my hs and reinstall it?

@tidylobster

Ilnur Garifullin

@tidylobster

Tags shouldn't affect very much. Runtimes are pretty stable right now, and we just update them to keep in sync with the main version of Hydrosphere.

@KineticCookie do you have any ideas, why @danikgan is receiving 504?

Bulat

@KineticCookie

Hey @danikgan !
Could you please elaborate when do you encounter 504? What http endpoint returns the error?

Daniil Gannota

@danikgan

@KineticCookie sorry for late reply, didn't get a notification of your message.

Frankly, I am not entirely sure what's causing the error. I am running 'sudo hs -v upload'. Usually, I am on a bank's VPN but I discounted from them. Not sure what's causing the issue.

disconnected*

I even reinstall hs for the sack of reinstalling -- didn't help

although when I run hs--version I get 2.2.0 not 2.2.1

Daniil Gannota

@danikgan

http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 504 173

Daniil Gannota

@danikgan

PS Build logs don't show up anything as well

Bulat

@KineticCookie

sorry for late reply, didn't get a notification of your message.

Yeah, sometimes gitter does that. I didn't get them either ¯_(ツ)_/¯

There is possibility that the infrastructure didn't start up properly, and nginx can't pass requests.

Assuming that you deployed hydroserving on docker-compose, could you please send the result of docker ps -a with hydroserving containers and logs of postgres manager and managerui containers

@danikgan ping just in case gitter sends notification to the void

Daniil Gannota

@danikgan

@KineticCookie in case you like screenshots more:

Daniil Gannota

@danikgan

f9317f35aed2 f5895cade5d0 "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago angry_lederberg
82059c92a976 164367ad044d "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago objective_mcclintock
339e4f6ac31e 359e7316378e "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago musing_torvalds
d89f5281975e 3c997be69bd0 "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago upbeat_sinoussi
217b0f30723a 2241f577786f "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago keen_satoshi
d26cc60be162 hydrosphere/serving-manager-ui:2.2.0 "/bin/sh -c 'envsubs…" 3 weeks ago Up 45 seconds 0.0.0.0:80->80/tcp managerui
c450de23219e hydrosphere/serving-gateway:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds gateway
5088293cf1f4 hydrosphere/serving-manager:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds 0.0.0.0:19090->9090/tcp, 0.0.0.0:19091->9091/tcp manager
ef61cd999b3e postgres:9.6-alpine "docker-entrypoint.s…" 3 weeks ago Up 46 seconds postgres

This is the output after a restart of my laptop and an opening of docker and minikube.

Daniil Gannota

@danikgan

@KineticCookie @tidylobster hi there!

Bulat

@KineticCookie

Hi @danikgan
Sorry for the delay. Didn't get the notification 😕

Could you also send the result of docker logs d26cc60be162 and docker logs 5088293cf1f4

Daniil Gannota

@danikgan

I get a huge output because I ran Hydrosphere too many time, so let me try to send it as a text file

I don't think you'll need all of it, but I am sending anyways just in case

docker logs d26cc60be162.txt

docker logs 5088293cf1f4.txt

Where communities thrive

People

Repo info

Activity