This error occured because manager service encountered some issue while processing you model files. To dig deeper into it, we'll need to look for manager service logs - they should contain progress of model processing.
In addition to that, in the latest dev version we implemented logs passthrough, so they'll be available to you via CLI or web UI. We'll release it as soon as we stabilize all the services. Looking forward to notify you when it's done! :)
If you have configured docker proxy at
~/.docker/config.json you can just mount this file to
/root/.docker/config.json inside the manager container.
Otherwise you can set it up using this docker documentation -> https://docs.docker.com/network/proxy/ and mount the configuration file.
@maikefischer sorry for the late reply - didn't get the notification :(
it' depends on your cluster deployment method. e.g. you can edit your docker-compose file to mount the file. With kubernetes you can use ConfigMap mount in deployment.
Sorry for the late reply. Monitoring service is not available as OSS right now, it's an enterprise feature. If you want, we can make a demo for you showing off monitoring service with various metrics, data profiling, reqstore. Please, submit a request via site https://hydrosphere.io/ for a more detailed discussion.
can i suggest updating https://hydrosphere.io/serving/ ? It says "Automated versioning, monitoring and testing functionality are included." and "Everything is open source. No premium features and enterprise versions. No hidden costs and learning curve can be as smooth as it gets."
I am trying to finish this tutorial: https://hydrosphere.io/serving-docs/latest/getting-started/serving-simple-model.html, but I have the following error:
sorry for late reply, didn't get a notification of your message.
Yeah, sometimes gitter does that. I didn't get them either ¯_(ツ)_/¯
There is possibility that the infrastructure didn't start up properly, and nginx can't pass requests.
Assuming that you deployed hydroserving on docker-compose, could you please send the result of
docker ps -a with hydroserving containers and logs of
f9317f35aed2 f5895cade5d0 "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago angry_lederberg 82059c92a976 164367ad044d "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago objective_mcclintock 339e4f6ac31e 359e7316378e "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago musing_torvalds d89f5281975e 3c997be69bd0 "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago upbeat_sinoussi 217b0f30723a 2241f577786f "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago keen_satoshi d26cc60be162 hydrosphere/serving-manager-ui:2.2.0 "/bin/sh -c 'envsubs…" 3 weeks ago Up 45 seconds 0.0.0.0:80->80/tcp managerui c450de23219e hydrosphere/serving-gateway:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds gateway 5088293cf1f4 hydrosphere/serving-manager:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds 0.0.0.0:19090->9090/tcp, 0.0.0.0:19091->9091/tcp manager ef61cd999b3e postgres:9.6-alpine "docker-entrypoint.s…" 3 weeks ago Up 46 seconds postgres
I have just checked your logs
Thread starvation or clock leap detected.
Seems like there are not enough resources for containers and it keeps failing because of connection leaks
Could you try to allocate more resources to the containers and check if it fixes the issue?