Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Daniil Gannota
    @danikgan
    This is what I get at the end of hs -v upload
    @akastav @KineticCookie
    Bulat
    @KineticCookie

    🤔
    Seems like it can't install dependency from requirements.txt

    Do you have some specific network settings, which could deny egress traffic from contrainers? Or you installed hydroserving from docker-compose.yaml as-is?
    People encountered similar issues, and seems like it's related to dns lookup on a host with docker daemon: https://forums.docker.com/t/facing-issue-with-creating-angular-application-image-using-docker/87270

    You could troubleshoot dns lookup from within a container in your machine. If the issue still persists after it, we'll try to come up with some fix/workaround.
    Daniil Gannota
    @danikgan

    Yes, I do indeed have a custom network setting. If it can't install the dependencies, might this mean an issue with pip?

    And yes, I installed hydro serving from docker-compose.yaml as is. I tried to configure it to use my pip setting: under docker-compose.yaml I have this line:

        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - ./pip:/root/.pip
    Under .pip I have the correct pip.conf and certificate from my workplace.
    But when I run docker-compose up and connect to the shell of manager container, I find that hydrosphere does not have pip installed.
    Daniil Gannota
    @danikgan
    @KineticCookie @akastav
    Ilnur Garifullin
    @tidylobster

    @danikgan You're on a right track, but you don't need to manipulate docker-compose.yaml to deliver your pip.conf file to the build context. One way of doing that might be just to include that file in the serving.yamlfile's payload and update install-comand (which is just a shell script). If we're talking about our linear_regression example, then it will look like this:

    kind: Model
    name: linear_regression
    runtime: "hydrosphere/serving-runtime-python-3.6:2.2.1"
    install-command: "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt"
    payload:
      - "src/"
      - "requirements.txt"
      - "pip.conf" # specify location of your pip.conf 
      - "model.h5"
    
    contract:
      name: infer
      inputs:
        x:
          shape: [-1, 2]
          type: double
          profile: numerical
      outputs:
        y:
          shape: [-1]
          type: double
          profile: numerical

    In this case I've just specified that pip.conf file lies within my current build context. So, pip should utilize pip.conf and you won't see Retrying (Retry(total=1, connect=None... during hs -v upload.

    If you don't want to put pip.conf into payload of your model, you can update install-command pip install -r requirements.txt to include all network settings required for your environment. For instance, pip install --proxy <proxy> -r requirements.txt.
    Daniil Gannota
    @danikgan

    Alright, we're on the right track -- that's great! Here's what I tried doing:

    1. Edited the serving.yaml file as specified.
    2. Added a certificate to the payload (as I need it for pip)
    3. Specified the location of the certificate under pip.conf file as cert = terra_bin.cert
    4. Turned on my VPN as it requires access to my workplace internet

    I still get an error though (and I tried various combinations I could think of to fix this). Could it be an issue with accepting the certificate? The pip.conf file should be correct as this is the same one I am using for my own local pip configuration (which works btw).

    Here's the output I get:
    debug: Current cluster: {'cluster': {'server': 'http://localhost'}, 'name': 'local'}
    debug: Payload src/ is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
    debug: Payload requirements.txt is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
    debug: Payload pip.conf is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
    debug: Payload tecca_bin.cer is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
    debug: Payload model.h5 is resolved as /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Popen(['git', 'version'], cwd=/Users/danikgan/Desktop/Hydrosphere/linear_regression, universal_newlines=False, shell=None, istream=None)
    debug: Error while extracting .git metadata: /Users/danikgan/Desktop/Hydrosphere/linear_regression
    debug: Can't extract DVC metadata: No module named 'dvc'
    Parsed model definition
    Name: linear_regression
    Runtime: hydrosphere/serving-runtime-python-3.7:2.2.0
    Install command: PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt
    Host selector: None
    Training data: None
    Signature name: infer
    Inputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | x      | [-1, 2] | DT_DOUBLE | NUMERICAL |
    Outputs:
    | name   | shape   | type      | profile   |
    |--------|---------|-----------|-----------|
    | y      | [-1]    | DT_DOUBLE | NUMERICAL |
    Monitoring:
        None
    Metadata:
    Payload:
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/src
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer
        /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5
    ================================
    debug: Creating archive: /Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/src as src
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/requirements.txt as requirements.txt
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/pip.conf as pip.conf
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/tecca_bin.cer as tecca_bin.cer
    debug: Archiving /Users/danikgan/Desktop/Hydrosphere/linear_regression/model.h5 as model.h5
    Uploading model to http://localhost
    ================================
    debug: MULTIPART POST: http://localhost/api/v2/model/upload. Parts: {'metadata': '{"contract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"name": "x", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}], "outputs": [{"name": "y", "profile": "NUMERICAL", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}, "dtype": "DT_DOUBLE"}]}}, "hostSelectorName": null, "runtime": {"tag": "2.2.0", "name": "hydrosphere/serving-runtime-python-3.7"}, "name": "linear_regression", "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "metadata": {}}', 'payload': ('filename', <_io.BufferedReader name='/Users/danikgan/Desktop/Hydrosphere/linear_regression/.hs/linear_regression/linear_regression.tar.gz'>)}
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/11/logs HTTP/1.1" 504 173
    error: Got server error <Response [504]>
    Error: Server returned an error: <html>
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</h1></center>
    <hr><center>openresty/1.15.8.3</center>
    </body>
    </html>
    I get the output above for tag 2.2.0. For tag 2.2.1 I get this ending:
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 200 690
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/12/logs HTTP/1.1" 404 74
    warning: Build logs are not available
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "GET /api/v2/model/version/linear_regression/12 HTTP/1.1" 200 742
    error: Model build failed
    {"model": {"id": 1, "name": "linear_regression"}, "image": {"name": "linear_regression", "tag": "12"}, "installCommand": "PIP_CONFIG_FILE=pip.conf pip install -r requirements.txt", "finished": "2020-06-03T11:25:24.405Z", "modelContract": {"modelName": "model", "predict": {"signatureName": "infer", "inputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "x", "shape": {"dim": [{"size": -1, "name": ""}, {"size": 2, "name": ""}], "unknownRank": false}}], "outputs": [{"profile": "NUMERICAL", "dtype": "DT_DOUBLE", "name": "y", "shape": {"dim": [{"size": -1, "name": ""}], "unknownRank": false}}]}}, "id": 12, "status": "Failed", "metadata": {}, "kind": "Internal", "modelVersion": 12, "runtime": {"name": "hydrosphere/serving-runtime-python-3.7", "tag": "2.2.1"}, "created": "2020-06-03T11:25:19.746Z"}
    Same outputs go for hydrosphere/serving-runtime-python-3.6, respectively.
    Any thoughts? @tidylobster @KineticCookie @akastav
    Ilnur Garifullin
    @tidylobster
    Hi @danikgan
    Sorry for late reply. What path are you using in your pip.conf to point to your certificate?
    Ilnur Garifullin
    @tidylobster
    If you place both pip.conf and your certificate (let's say cert.pem) into the payload within serving.yaml, they will be available under /model/files/pip.conf, /model/files/cert.pem paths. So, you might try to update your pip.conf to use the correct path to your cert.pem.
    Daniil Gannota
    @danikgan

    Unfortunately, this didn't help:

    1. I change the pip config file to cert=/model/files/cert.pem
    2. Launched VPN and all the rest is the same as described above

    Shall I provide you with some logs or additional context?
    @tidylobster @KineticCookie @akastav

    This is my current error:
    debug: Starting new HTTP connection (1): localhost:80
    debug: http://localhost:80 "POST /api/v2/model/upload HTTP/1.1" 504 173
    error: Got server error <Response [504]>
    Error: Server returned an error: <html>
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</h1></center>
    <hr><center>openresty/1.15.8.3</center>
    </body>
    </html>
    Ilnur Garifullin
    @tidylobster
    I'm not sure why we are back on 504 error. The server didn't even start building the model. Can this be because of low resources, as @KineticCookie mentioned here? https://gitter.im/Hydrospheredata/hydro-serving?at=5ec7c91ef0b8a2053aa80284
    Ilnur Garifullin
    @tidylobster
    So, did you change your settings for docker daemon from last try?
    Daniil Gannota
    @danikgan
    You're right!:) Finally, the sacred words of Success: {"model": {...
    Ilnur Garifullin
    @tidylobster
    Great! I'm glad it worked.
    gaopan1996
    @gaopan1996

    Hi, all. I have started the quickstart documentation for the hs cli on my Windows10 machine. I installed hs==2.4.0 and am getting this error

    hs --version
    Traceback (most recent call last):
        File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
            "__main__", mod_spec)
        File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
            exec(code, run_globals)
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\Scripts\hs.exe\__main__.py", line 5, in <module>
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\commands\__init__.py", line 1, in <module>
            from hydroserving.cli.commands.hs import * 
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\commands\hs.py", line 7, in <module>
            from hydroserving.cli.context_object import ContextObject
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\cli\context_object.py", line 7, in <module>
            from hydroserving.core.apply import ApplyService
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\core\apply.py", line 8, in <module>
            from hydroserving.core.deployment_config.parser import parse_deployment_configuration
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydroserving\core\deployment_config\parser.py", line 3, in <module>
            from hydrosdk import DeploymentConfiguration
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\__init__.py", line 3, in <module>
            from hydrosdk.modelversion import LocalModel, ModelVersion
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\modelversion.py", line 19, in <module>
            from hydrosdk.contract import ModelContract_to_contract_dict, contract_dict_to_ModelContract, validate_contract
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\contract.py", line 11, in <module>
            from hydrosdk.data.types import alias_to_proto_dtype, shape_to_proto, PY_TO_DTYPE, np_to_proto_dtype, proto_to_np_dtype
        File "C:\Users\K27793\AppData\Roaming\Python\Python36\site-packages\hydrosdk\data\types.py", line 130, in <module>
        np.float128: None,
    AttributeError: module 'numpy' has no attribute 'float128'

    which is related to this issue winpython/winpython#613.
    Seems like an issue with hydro-sdk on this line https://github.com/Hydrospheredata/hydro-serving-sdk/blob/master/hydrosdk/data/types.py#L130

    Downgrading to hs==2.2.0, which doesn't have hydro-sdk dependency, worked on Windows10.

    Yuriy
    @Valenzione

    Hi @gaopan1996 , thanks for noticing this issue! We indeed do not perform multi-platform testing of our client libraries, so we are mostly unaware of possible issues for our Windows users.

    Since this is an issue, I suggest you posting the same text you’ve sent us here to our Github issues via this link: https://github.com/Hydrospheredata/hydro-serving/issues/new

    I'll return shortly after with a solution and post it directly to the issue you will have created.

    gaopan1996
    @gaopan1996
    Thanks @Valenzione. Created issue here Hydrospheredata/hydro-serving#330