h2oai/h2o-3

Fast Scalable Machine Learning API For Smarter Applications

benoit

@BenoitLondon

i have installed h2o in R from here http://h2o-release.s3.amazonaws.com/h2o/master/4919/index.html

if I exclude XGBooost it works fine

benoit

@BenoitLondon

even with a small dataset I get this issue so I believe it s not a RAM issue

razou

@razou

Hi I’m using H2O XGBoost with following settings:

Cluster size : 3 nodes and 800 GB of memory
Data: 50 M rows and 15 columns
XGBoost: max_depth=6, ntrees=200, subsample=0.8, eta=0.05

I found it extremely slow compare to other xgboost’s implementations (e.g dmlc): more than 3 hours
Is it normal ?

Thanks

Michal Kurka

@michalkurka

are you comparing distributed impl to a single node impl?

or is it multinode-multinode comparison?

(for a dataset like yours, a single node should be just fine)

razou

@razou

I tested on single node with 400GB of memory but It was also very slow

Michal Kurka

@michalkurka

ok, our benchmarks don't show any such issue; that makes me think it must be something about your setup

What is your H2O version and what does H2O prints when its starts - it should tell you what XGBoost backend is being used, you want to make sure you are using OpenMP, not the single-threaded fallback version.

razou

@razou

I using h2o-3.28.0.1
I’m h2o via jupyter notebook on AWS cloud and

The way I’m installing h2o: conda install -c h2oai h2o

Here is my code
The output ou H2O start

Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "11.0.5" 2019-10-15; OpenJDK Runtime Environment (build 11.0.5+10-post-Ubuntu-0ubuntu1.118.04); OpenJDK 64-Bit Server VM (build 11.0.5+10-post-Ubuntu-0ubuntu1.118.04, mixed mode, sharing)
  Starting server from /opt/conda/lib/python3.6/site-packages/h2o/backend/bin/h2o.jar
  Ice root: /tmp/tmptc1_6xe2
  JVM stdout: /tmp/tmptc1_6xe2/h2o_jovyan_started_from_python.out
  JVM stderr: /tmp/tmptc1_6xe2/h2o_jovyan_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321 ... successful.
H2O cluster uptime:    01 secs
H2O cluster timezone:    Etc/GMT
H2O data parsing timezone:    UTC
H2O cluster version:    3.28.0.1
H2O cluster version age:    1 month
H2O cluster name:    H2O_from_python_jovyan_fmr3t8
H2O cluster total nodes:    1
H2O cluster free memory:    400 Gb
H2O cluster total cores:    64
H2O cluster allowed cores:    64
H2O cluster status:    accepting new members, healthy
H2O connection url:    http://127.0.0.1:54321
H2O connection proxy:    {'http': None, 'https': None}
H2O internal security:    False
H2O API Extensions:    Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4
Python version:    3.6.8 final

Youshaa Murhij

@YoushaaMurhij

I am using Driverless AI inside Docker on Windows 10 Pro .. but the problem is that I can not access it through localhost:12345 .. anyone faced the same problem?

razou

@razou

@YoushaaMurhij same problem on AWS where I’m running h2o is running inside docker container

Michal Malohlava

@mmalohlava

@YoushaaMurhij @razou did you expose the port 12345 via docker -p option?
@razou Please also make sure that you security group assigned to your instance allows TCP port 12345?

btw: there is a Slack community where you can ask Driverless AI related questions: https://www.h2o.ai/slack-community/

razou

@razou

Thanks @mmalohlava
My security group allows the port 12345 but I’ don’t know if the port was exposed, i’ll check this

razou

@razou

Hi, does anybody faced this version mismatch error before ?

pip uninstall h2o
pip install https://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/Python/h2o-3.28.0.2-py2.py3-none-any.whl

import h2o
h2o.init()

I’m getting this error

H2OConnectionError: Version mismatch. H2O is version 3.28.0.2, but the h2o-python package is version 3.28.0.1. Install the matching h2o-Python version from - http://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/index.html.

Honza Sterba

@honzasterba

@razou that is a strange behaviour, what is the output of h2o.init()?

razou

@razou

the output of h2o.init() is the error above

Honza Sterba

@honzasterba

in the output above I can see " to H2O server at http://127.0.0.1:54321 ... successful" and no H2OConnectionError, what am I missing?

razou

@razou

oh, that’s two different things, when I say above I mean this one:

H2OConnectionError: Version mismatch. H2O is version 3.28.0.2, but the h2o-python package is version 3.28.0.1. Install the matching h2o-Python version from - http://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/index.html.

only my last post (starting from pip uninstall h2o)

angela0xdata

@angela0xdata

Can you launch Flow and make sure that it was shut down properly?

razou

@razou

It’s always an EMR new cluster different from the preious on (not same machines)

Michal Kurka

@michalkurka

@razou the python package is built correctly (I verified that), can you provide the full output that ends with "Version mismatch” error?

Jacob Niebloom

@niebloomj

does anyone know anything about this error? I am running h2o on emr with spark. Thank you!

01-22 20:14:33.229 17479  #59:54321 ERRR: java.lang.ArrayIndexOutOfBoundsException: 13
01-22 20:14:33.229 17479  #59:54321 ERRR:     at water.TCPReceiverThread.basic_packet_handling(TCPReceiverThread.java:307)
01-22 20:14:33.229 17479  #59:54321 ERRR:     at water.TCPReceiverThread$UDP_TCP_ReaderThread.run(TCPReceiverThread.java:249)
01-22 20:14:33.236 17479  #18:54321 WARN: UDP packets from outside the cloud: 1/sec, last one from /:54321 @ Wed Jan 22 20:14:33 UTC 2020
01-22 20:14:33.236 17479  #83:54321 WARN: UDP packets from outside the cloud: 2/sec, last one from /:54321 @ Wed Jan 22 20:14:33 UTC 2020
01-22 20:14:33.240 17479  #69:54321 ERRR: java.lang.ArrayIndexOutOfBoundsException: 13
01-22 20:14:33.240 17479  #69:54321 ERRR:     at water.TCPReceiverThread.basic_packet_handling(TCPReceiverThread.java:307)
01-22 20:14:33.240 17479  #69:54321 ERRR:     at water.TCPReceiverThread$UDP_TCP_ReaderThread.run(TCPReceiverThread.java:249)
01-22 20:14:33.241 17479  #82:54321 ERRR: java.lang.ArrayIndexOutOfBoundsException: 16
01-22 20:14:33.241 17479  #82:54321 ERRR:     at water.TCPReceiverThread.basic_packet_handling(TCPReceiverThread.java:307)
01-22 20:14:33.241 17479  #82:54321 ERRR:     at water.TCPReceiverThread$UDP_TCP_ReaderThread.run(TCPReceiverThread.java:249)
01-22 20:14:33.242 17479  #50:54321 ERRR: java.lang.UnsupportedOperationException
01-22 20:14:33.242 17479  #50:54321 ERRR:     at java.nio.ByteBuffer.array(ByteBuffer.java:994)
01-22 20:14:33.242 17479  #50:54321 ERRR:     at water.H2ONode$UDP_TCP_SendThread.run(H2ONode.java:556)

razou

@razou

@michalkurka @honzasterba here is the complete output https://imgur.com/284Oovc

razou

@razou

@michalkurka @honzasterba I don’t know how and why, but the problem disappears (today, on new emr cluster) and this the following works correctly

pip uninstall h2o
pip install https://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/Python/h2o-3.28.0.2-py2.py3-none-any.whl

if h2o.cluster():
    h2o.cluster().shutdown()

h2o.init()

Matthew J. Oldach

@moldach

Hello, I'm not sure if this is best posted here or as an issue in one of the h20ai github repos (if so please let me know where the best place to post question related to H2oai for R).

I'm trying to get the leaf indices that each sample was assigned to in the forest from h2o.randomForest() (as one can get from sklearn's ExtraTreesClassifier or from R's randomForest and ranger).

library(h2o)
h2o.init(nthreads = 1)

# convert data to h2o objects
train <- as.h2o(d_train)
test <- as.h2o(d_test)

# Convert response to factor (required by randomForest)
train[,ycol] <- as.factor(train[,ycol])
test[,ycol] <- as.factor(test[,ycol])

system.time(
  model <- h2o.randomForest(
    x = xcols,
    y = ycol,
    training_frame = train,
    seed = 123,
    ntrees = 100  ### which parameter do I need to get leaf indices assigned to each sample?
    )
  )

I had made a more detailed post on StackOverflow but haven't gotten any feedback so I wanted to reach out directly to the devs.

Thank you

razou

@razou

@michalkurka @honzasterba I don’t know how and why, but the problem disappears (today, on new emr cluster) and this the following works correctly
pip uninstall h2o
pip install https://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/Python/h2o-3.28.0.2-py2.py3-none-any.whl
if h2o.cluster():
    h2o.cluster().shutdown()

h2o.init()

I’m getting this error again

Checking whether there is an H2O instance running at http://127.0.0.1:54321 ..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "1.8.0_232"; OpenJDK Runtime Environment (build 1.8.0_232-b09); OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
  Starting server from /usr/local/lib/python3.6/site-packages/h2o/backend/bin/h2o.jar
  Ice root: /tmp/tmp7mjsqtkb
  JVM stdout: /tmp/tmp7mjsqtkb/h2o_hadoop_started_from_python.out
  JVM stderr: /tmp/tmp7mjsqtkb/h2o_hadoop_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321 ... successful.
Traceback (most recent call last):
  File "/mnt/tmp/spark-354a5e4e-ea63-486d-8ec4-20d78a6749a4/adx_floor_train_4_0.py", line 428, in <module>
    TrainGBM()()
  File "/mnt/tmp/spark-354a5e4e-ea63-486d-8ec4-20d78a6749a4/adx_floor_train_4_0.py", line 364, in __call__
    h2o.init(min_mem_size='60G', ip='127.0.0.1', port=54321)
  File "/usr/local/lib/python3.6/site-packages/h2o/h2o.py", line 318, in init
    version_check()
  File "/usr/local/lib/python3.6/site-packages/h2o/h2o.py", line 165, in version_check
    "".format(ver_h2o, ver_pkg, branch_name_h2o, build_number_h2o))
h2o.exceptions.H2OConnectionError: Version mismatch. H2O is version 3.28.0.2, but the h2o-python package is version 3.28.0.3. Install the matching h2o-Python version from - http://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/index.html.
Closing connection _sid_b1b9 at exit
H2O session _sid_b1b9 closed.

Richard V.

@Rokko11

Hello! I try to automate the execution of a .flow-file in H2O Flow. Do you know if there is a REST command for the "Run all" Command?

Mauricio Alarcon

@rmalarc

hi there!. I'm having an issue with a DF where h2o seems to present the wrong datatype. In this case, it's zero inflated (1.7M zeros/1.8M total nrows).

print(df.types[var])
bool

however describe shows the correct datatype (real):

df[var].describe()
         promoted_event_sales_fixed
-------  ----------------------------
type     real
mins     0.0
mean     1.0572684333157962
maxs     216.0
sigma    5.052133552611355
zeros    1714635
missing  0
0        0.0
1        0.0
2        0.0
3        0.0
4        0.0
5        0.0
6        0.0
7        0.0
8        0.0
9        0.0

At this point, I'm unable to do much with this variable (can't calculate means, sd, etc), these functions think it's a boolean

H2O cluster version:        3.28.0.3

any ideas on how to get around this?

Mauricio Alarcon

@rmalarc

it seems to get parsed correctly once imported from the file, but it somehow changes to bool once columns are sliced

Michal Kurka

@michalkurka

At this point, I'm unable to do much with this variable (can't calculate means, sd, etc), these functions think it's a boolean

H2O on the backend doesn't actually recognize a "boolean" type (it only has numbers, categoricals, strings and uuid). It is the python client that guesses it would be a boolean.

This is a bug, I will bring this to the attention of the package maintainer

Seb

@sebhrusen

@rmalarc can you please show how this df was obtained? was it first created from Python? basically I'd like to know all the transformations on the df (if any) from its initial upload to the one with the wrong column type.
It would help me to reproduce the issue.

Sandeep Srinivasa

@sandys

hi guys, how do i test out driverless ai ? i cant seem to figure out how to download/test, etc

Zachary Raicik

@zakraicik

Does anyone know how to transform log odds explanations into probability explanations? It can be done easily in the SHAP package but it's not clear how to use the h2o predict_contributions functions and transform it.

Simon Schmid

@SimonSchmid

Hi,
we encounter some issue running h2o on WindowsServer. When H2O is started, the following exception occurrs:

java.lang.RuntimeException: java.net.SocketException: Unrecognized Windows Sockets error: 0: Cannot bind
    at water.MultiReceiverThread.run(MultiReceiverThread.java:76)
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: Cannot bind
    at java.net.TwoStacksPlainDatagramSocketImpl.bind0(Native Method)
    at java.net.TwoStacksPlainDatagramSocketImpl.bind0(TwoStacksPlainDatagramSocketImpl.java:107)
    at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:93)
    at java.net.TwoStacksPlainDatagramSocketImpl.bind(TwoStacksPlainDatagramSocketImpl.java:97)
    at java.net.DatagramSocket.bind(DatagramSocket.java:392)
    at java.net.MulticastSocket.<init>(MulticastSocket.java:172)
    at java.net.MulticastSocket.<init>(MulticastSocket.java:136)
    at water.MultiReceiverThread.run(MultiReceiverThread.java:55)

However, H2O still seems to work afterwards. Can someone give me a hint what this exception means/why it occurrs and if it is actually harmless?
(Version 3.24.0.4, i.e., line 55 here https://github.com/h2oai/h2o-3/blob/017314aec9eeb5459727d0e0047082cc23e9d298/h2o-core/src/main/java/water/MultiReceiverThread.java)

Simon Schmid

@SimonSchmid

We are running h2o local, should we actually use flatfile instead of multicast?

lohralexander

@lohralexander

Hi, does anyone know why there is no pojo export for isolation forests?

Jordan Bentley

@jbentleyEG

H2O is an entirely remote company right? Any good tips for the rest of us suddenly finding ourselves in that position?

Honza Sterba

@honzasterba

@jbentleyEG normally we are far from being entirely remote, but the last two weeks we have been. So even for many people in H2O this is a new experience.

@SimonSchmid by local you mean on single machine? using flatfile is generally done when multicast does not work.

@SimonSchmid looks like a JVM issue and not H2O bug. this looks like a promissing thread https://stackoverflow.com/questions/3947555/java-net-socketexception-unrecognized-windows-sockets-error-0-jvm-bind-jboss

Zachary Raicik

@zakraicik

Does anyone know how to transform log odds explanations into probability explanations? It can be done easily in the SHAP package but it's not clear how to use the h2o predict_contributions functions and transform it.

Does this mean no one knows?

Simon Schmid

@SimonSchmid

Hi @honzasterba thanks for the response. Yes, I mean on a single machine. Multicast is working in general (besides the Windows Sockets error). Do you we have any disadvantage using fatfile? This may be the easist fix. We could also just catch the exception and ignore it since h2o still seems to work afterwards. We already looked into the thread you posted but the proposed solutions did not help us. That's why we hoped to find a solution here. However, if this is not a familiar issue for you, we may need to find the causing issue ourselves, thanks for the help anyway!

Honza Sterba

@honzasterba

@SimonSchmid any reason why are you running multiple nodes on a single machine? it definitely more effective to run one big node on one machine, multinode is useful when you want to scale horizontally across multiple machines

There is no disadvantage to using flatfile.

Cheng WeI

@valkyrias_gitlab

Hi All, I am wondering if there is any support document for using H2O model for real-time scoring? Anything will help. Thank you!

Jakub Háva

@jakubhava

@valkyrias_gitlab one way how to score H2O models might be using Sparkling Water, please check this link http://docs.h2o.ai/sparkling-water/2.4/latest-stable/doc/deployment/load_mojo.html

Simon Schmid

@SimonSchmid

@honzasterba Thanks for the clarifcation, I think I missunderstood the concept. Sorry for the confusion! I thought flatfile mode means running a single node. So currently we are starting h2o with:

H2OStarter.start(new String[]{"-name", cloudName, "-disable_web", "-nthreads", "" + numThreads, "-port",
            "" + port, "-quiet", "-ice_root", path}, false);

This should actually just start one node, right? Is there a way to tell h2o that no furhter node will be setup, so that it does not even run into line 55 in MultiReceiverThread and no exception is thrown? Since there is in line 27 if (H2O.isFlatfileEnabled()) return;, I assumed using flatfile could solve my issue. Any ideas?

Cheng WeI

@valkyrias_gitlab

@jakubhava Thanks for the reply. I used SparklingWater to train the model and created a MOJO file already. However it will be great if there is document regarding how to to a real-time scoring so that i can support our data engineer better in deployment.

Honza Sterba

@honzasterba

@SimonSchmid yes, that starts one node. To set flatfile mode you need to add -flatfile=path/to/flatfile.txt. The flatfile.txt can actually be an empty file, you will get a warning about that, but it has no effect.

Where communities thrive

People

Repo info

Activity