Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Daniil Gannota
    @danikgan
    I even reinstall hs for the sack of reinstalling -- didn't help
    although when I run hs--version I get 2.2.0 not 2.2.1
    Daniil Gannota
    @danikgan
    http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 504 173
    Daniil Gannota
    @danikgan
    PS Build logs don't show up anything as well
    Bulat
    @KineticCookie

    sorry for late reply, didn't get a notification of your message.

    Yeah, sometimes gitter does that. I didn't get them either ¯_(ツ)_/¯

    There is possibility that the infrastructure didn't start up properly, and nginx can't pass requests.

    Assuming that you deployed hydroserving on docker-compose, could you please send the result of docker ps -a with hydroserving containers and logs of postgres manager and managerui containers

    @danikgan ping just in case gitter sends notification to the void
    Daniil Gannota
    @danikgan
    @KineticCookie in case you like screenshots more:
    Screenshot 2020-05-18 at 17.23.56.png
    Daniil Gannota
    @danikgan
    f9317f35aed2 f5895cade5d0 "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago angry_lederberg
    82059c92a976 164367ad044d "/bin/sh -c 'pip ins…" 2 days ago Exited (1) 2 days ago objective_mcclintock
    339e4f6ac31e 359e7316378e "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago musing_torvalds
    d89f5281975e 3c997be69bd0 "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago upbeat_sinoussi
    217b0f30723a 2241f577786f "/bin/sh -c 'pip ins…" 4 days ago Exited (1) 4 days ago keen_satoshi
    d26cc60be162 hydrosphere/serving-manager-ui:2.2.0 "/bin/sh -c 'envsubs…" 3 weeks ago Up 45 seconds 0.0.0.0:80->80/tcp managerui
    c450de23219e hydrosphere/serving-gateway:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds gateway
    5088293cf1f4 hydrosphere/serving-manager:2.2.0 "/hydro-serving/app/…" 3 weeks ago Up 46 seconds 0.0.0.0:19090->9090/tcp, 0.0.0.0:19091->9091/tcp manager
    ef61cd999b3e postgres:9.6-alpine "docker-entrypoint.s…" 3 weeks ago Up 46 seconds postgres
    This is the output after a restart of my laptop and an opening of docker and minikube.
    Daniil Gannota
    @danikgan
    @KineticCookie @tidylobster hi there!
    Bulat
    @KineticCookie

    Hi @danikgan
    Sorry for the delay. Didn't get the notification 😕

    Could you also send the result of docker logs d26cc60be162 and docker logs 5088293cf1f4

    Daniil Gannota
    @danikgan
    I get a huge output because I ran Hydrosphere too many time, so let me try to send it as a text file
    I don't think you'll need all of it, but I am sending anyways just in case
    Daniil Gannota
    @danikgan
    @KineticCookie hope this notification finds you!;)
    Daniil Gannota
    @danikgan
    SN Whenever I try to upload a model, it throws me either 504 gateway error or "model build failed" error after line "http://localhost:80 "GET /api/v2/model/version/linear_regression/24 HTTP/1.1" 200 717". I don't really know when it throws 504 and when build failure -- seems unpredictable to me.
    Bulat
    @KineticCookie

    @danikgan hi!

    I have just checked your logs

    Thread starvation or clock leap detected.

    Seems like there are not enough resources for containers and it keeps failing because of connection leaks

    Could you try to allocate more resources to the containers and check if it fixes the issue?

    Daniil Gannota
    @danikgan
    What would be the minimal requirements to run hs?
    @KineticCookie
    Daniil Gannota
    @danikgan

    I allocated 8GB to Docker and 4 cpus.
    I allocated the same for minikube which I use for K8S.

    I'll send you the latest logs now, but do you still think it's to do with hs running out of resources?

    Daniil Gannota
    @danikgan
    debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
    debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/src
    debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/increment_model, universal_newlines=False, shell=None, istream=None)
    debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/increment_model
    debug: Can't extract DVC metadata: No module named 'dvc'
    Parsed model definition
    Name: increment_model
    Runtime: hydrosphere/serving-runtime-python-3.6:2.2.1
    Install command: pip install -r requirements.txt
    Host selector: None
    Training data: None
    Signature name: increment
    Inputs:
    | name   | shape   | type     | profile   |
    |--------|---------|----------|-----------|
    | number | [-1]    | DT_INT32 | NUMERICAL |
    Outputs:
    | name   | shape   | type     | profile   |
    |--------|---------|----------|-----------|
    | number | [-1]    | DT_INT32 | NUMERICAL |
    Monitoring:
        None
    Metadata:
    Payload:
        /Users/danikgan/Desktop/Hydrosphere/increment_model/src
        /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt
    ================================
    debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/src as src
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/increment_model/requirements.txt as requirements.txt
    Uploading model to http://localhost
    ================================
    debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "increment", "inputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}], "outputs": [{"name": "number", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_INT32"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.1", "name": "hydrosphere/serving-runtime-python-3.6"}, "name": "increment_model", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/increment_model/.hs/increment_model/increment_model.tar.gz'>)}
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 650
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/73/logs HTTP/1.1" 200 None
    debug: Initialized SSE client from event source <Response [200]>
    Build logs:
    debug: Dispatching Log event #0, no data...
    
    debug: Dispatching Log event #1, 60 bytes...
    Step 1/6 : FROM hydrosphere/serving-runtime-python-3.6:2.2.1
    debug: Dispatching Log event #2, 103 bytes...
    manifest for hydrosphere/serving-runtime-python-3.6:2.2.1 not found: manifest unknown: manifest unknown
    debug: Dispatching EndOfStream event, no data...
    
    ================================
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/increment_model/7 HTTP/1.1" 200 702
    error: Model build failed
    debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
    debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
    debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
    debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
    debug: Can't extract DVC metadata: No module named 'dvc'
    Parsed model definition
    Name: linear_regression
    Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
    Install command: pip install -r requirements.txt
    Host selector: None
    Training data: None
    Signature name: infer
    Inputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | x      | [-1, 2] | DT_DOUBLE | NUMERICAL |
    Outputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | y      | [-1]    | DT_DOUBLE | NUMERICAL |
    Monitoring:
        None
    Metadata:
    Payload:
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    ================================
    debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
    Uploading model to http://localhost
    ================================
    debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 665
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/74/logs HTTP/1.1" 200 None
    debug: Initialized SSE client from event source <Response [200]>
    Build logs:
    debug: Dispatching Log event #0, no data...
    
    debug: Dispatching Log event #1, 60 bytes...
    Step 1/6 : FROM hydrosphere/serving-runtime-python-3.7:2.2.0
    debug: Dispatching Log event #2, 103 bytes...
    manifest for hydrosphere/serving-runtime-python-3.7:2.2.0 not found: manifest unknown: manifest unknown
    debug: Dispatching EndOfStream event, no data...
    
    ================================
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/35 HTTP/1.1" 200 717
    error: Model build failed
    Even trying to upload different ML models produces exactly the same procedure for error
    Bulat
    @KineticCookie
    Hi!
    Seems like a problem here is that docker can't find the image.
    I re-released the images, could you try again?
    Daniil Gannota
    @danikgan
    Okay, so I deleted the old hs images, installed the new ones but still the same error
    Could it be because I am on Helm 3? I need to find how to downgrade to v2.9 then
    Dmitriy Kharlamov
    @akastav
    Hi Daniil! Could you try this command? docker pull hydrosphere/serving-runtime-python-3.7:2.2.0
    Daniil Gannota
    @danikgan
    2.2.0: Pulling from hydrosphere/serving-runtime-python-3.7
    Digest: sha256:e62e47b3352b23788e4e31f3e4a5636328a17cc65e7d44b5e234a6f3868cd977
    Status: Image is up to date for hydrosphere/serving-runtime-python-3.7:2.2.0
    docker.io/hydrosphere/serving-runtime-python-3.7:2.2.0
    debug: Dispatching Log event #406, 10 bytes...
    Extracting
    debug: Dispatching Log event #407, 13 bytes...
    Pull complete
    debug: Dispatching Log event #408, 79 bytes...
    Digest: sha256:e62e47b3352b23788e4e31f3e4a5636328a17cc65e7d44b5e234a6f3868cd977
    debug: Dispatching Log event #409, 79 bytes...
    Status: Downloaded newer image for hydrosphere/serving-runtime-python-3.7:2.2.0
    debug: Dispatching Log event #410, 17 bytes...
    ---> 90f743fd43f9
    debug: Dispatching Log event #411, 45 bytes...
    Step 2/6 : LABEL MODEL_NAME=linear_regression
    debug: Dispatching Log event #412, 28 bytes...
    ---> Running in 1373c319bfc5
    debug: Dispatching Log event #413, 44 bytes...
    Removing intermediate container 1373c319bfc5
    debug: Dispatching Log event #414, 17 bytes...
    ---> dc43c044b0a8
    debug: Dispatching Log event #415, 32 bytes...
    Step 3/6 : LABEL MODEL_VERSION=2
    debug: Dispatching Log event #416, 28 bytes...
    ---> Running in 34b07afa09bb
    debug: Dispatching Log event #417, 44 bytes...
    Removing intermediate container 34b07afa09bb
    debug: Dispatching Log event #418, 17 bytes...
    ---> 8d29ff4fe4dc
    debug: Dispatching Log event #419, 27 bytes...
    Step 4/6 : ADD model /model
    debug: Dispatching Log event #420, 17 bytes...
    ---> bdeb5bca1ddb
    debug: Dispatching Log event #421, 31 bytes...
    Step 5/6 : WORKDIR /model/files
    debug: Dispatching Log event #422, 28 bytes...
    ---> Running in 9aaea0b13b3f
    debug: Dispatching Log event #423, 44 bytes...
    Removing intermediate container 9aaea0b13b3f
    debug: Dispatching Log event #424, 17 bytes...
    ---> a86bfaf28563
    debug: Dispatching Log event #425, 46 bytes...
    Step 6/6 : RUN pip install -r requirements.txt
    debug: Dispatching Log event #426, 28 bytes...
    ---> Running in 0aa6ee58f721
    debug: Dispatching message event, no data...
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching Log event #427, 332 bytes...
    [91mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef890a50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching Log event #428, 332 bytes...
    [91mWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7490>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching Log event #429, 332 bytes...
    [91mWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7590>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching Log event #430, 332 bytes...
    [91mWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7c90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching message event, no data...
    
    debug: Dispatching Log event #431, 332 bytes...
    [91mWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd4ef8a7dd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/keras/
    
    debug: Dispatching message event, no data.
    This is what I get at the end of hs -v upload
    @akastav @KineticCookie
    Bulat
    @KineticCookie

    🤔
    Seems like it can't install dependency from requirements.txt

    Do you have some specific network settings, which could deny egress traffic from contrainers? Or you installed hydroserving from docker-compose.yaml as-is?
    People encountered similar issues, and seems like it's related to dns lookup on a host with docker daemon: https://forums.docker.com/t/facing-issue-with-creating-angular-application-image-using-docker/87270

    You could troubleshoot dns lookup from within a container in your machine. If the issue still persists after it, we'll try to come up with some fix/workaround.
    Daniil Gannota
    @danikgan

    Yes, I do indeed have a custom network setting. If it can't install the dependencies, might this mean an issue with pip?

    And yes, I installed hydro serving from docker-compose.yaml as is. I tried to configure it to use my pip setting: under docker-compose.yaml I have this line:

        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - ./pip:/root/.pip
    Under .pip I have the correct pip.conf and certificate from my workplace.
    But when I run docker-compose up and connect to the shell of manager container, I find that hydrosphere does not have pip installed.
    Daniil Gannota
    @danikgan
    @KineticCookie @akastav
    Ilnur Garifullin
    @tidylobster

    @danikgan You're on a right track, but you don't need to manipulate docker-compose.yaml to deliver your pip.conf file to the build context. One way of doing that might be just to include that file in the serving.yamlfile's payload and update install-comand (which is just a shell script). If we're talking about our linear_regression example, then it will look like this:

    kind: Model
    name: linear_regression
    runtime: "hydrosphere/serving-runtime-python-3.6:2.2.1"
    install-command: "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt"
    payload:
      - "src/"
      - "requirements.txt"
      - "pip.conf" # specify location of your pip.conf 
      - "model.h5"
    
    contract:
      name: infer
      inputs:
        x:
          shape: [-1, 2]
          type: double
          profile: numerical
      outputs:
        y:
          shape: [-1]
          type: double
          profile: numerical

    In this case I've just specified that pip.conf file lies within my current build context. So, pip should utilize pip.conf and you won't see Retrying (Retry(total=1, connect=None... during hs -v upload.

    If you don't want to put pip.conf into payload of your model, you can update install-command pip install -r requirements.txt to include all network settings required for your environment. For instance, pip install --proxy <proxy> -r requirements.txt.
    Daniil Gannota
    @danikgan

    Alright, we're on the right track -- that's great! Here's what I tried doing:

    1. Edited the serving.yaml file as specified.
    2. Added a certificate to the payload (as I need it for pip)
    3. Specified the location of the certificate under pip.conf file as cert = terra_bin.cert
    4. Turned on my VPN as it requires access to my workplace internet

    I still get an error though (and I tried various combinations I could think of to fix this). Could it be an issue with accepting the certificate? The pip.conf file should be correct as this is the same one I am using for my own local pip configuration (which works btw).

    Here's the output I get:
    debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
    debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
    debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
    debug: Payload pip.conf is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
    debug: Payload tecca_bin.cer is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
    debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
    debug: Can't extract DVC metadata: No module named 'dvc'
    Parsed model definition
    Name: linear_regression
    Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
    Install command: PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt
    Host selector: None
    Training data: None
    Signature name: infer
    Inputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | x      | [-1, 2] | DT_DOUBLE | NUMERICAL |
    Outputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | y      | [-1]    | DT_DOUBLE | NUMERICAL |
    Monitoring:
        None
    Metadata:
    Payload:
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    ================================
    debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf as pip.conf
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer as tecca_bin.cer
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
    Uploading model to http://localhost
    ================================
    debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/11/logs HTTP/1.1" 504 173
    error: Got server error <Response [504]>
    Error: Server returned an error: <html>
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</h1></center>
    <hr><center>openresty/1.15.8.3</center>
    </body>
    </html>
    I get the output above for tag 2.2.0. For tag 2.2.1 I get this ending:
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/12/logs HTTP/1.1" 404 74
    warning: Build logs are not available
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/12 HTTP/1.1" 200 742
    error: Model build failed
    {"model": {"id": 1, "name": "linear_regression"}, "image": {"name": "linear_regression", "tag": "12"}, "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "finished": "2020-06-03T11:25:24.405Z", "modelContract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "x", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}}], "outputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "y", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}}]}}, "id": 12, "status": "Failed", "metadata": {}, "kind": "Internal", "modelVersion": 12, "runtime": {"name": "hydrosphere/serving-runtime-python-3.7", "tag": "2.2.1"}, "created": "2020-06-03T11:25:19.746Z"}
    Same outputs go for hydrosphere/serving-runtime-python-3.6, respectively.
    Any thoughts? @tidylobster @KineticCookie @akastav
    Ilnur Garifullin
    @tidylobster
    Hi @danikgan
    Sorry for late reply. What path are you using in your pip.conf to point to your certificate?
    Ilnur Garifullin
    @tidylobster
    If you place both pip.conf and your certificate (let's say cert.pem) into the payload within serving.yaml, they will be available under /model/files/pip.conf, /model/files/cert.pem paths. So, you might try to update your pip.conf to use the correct path to your cert.pem.