Hi,
I’m guetting strange behaviors when reading data (folder containing many csv files) from Amazon S3 with import_file
- I generate my data with Spark + Scala and write it into csv format in an S3 folder
- When loading that data with
h2o.import_file(path=s3_path, pattern=".*\.csv”)I’m gueting some data of thecolumn 28int thecolumn 1and some data of thecolumn 29int thecolumn 2, …
Does this function have a limitation in number of columns or did I missided on an addtional option, or it’s due to something else ?
Thanks
9 replies
Hi
I wanted to perform a frequency encoding on categorical features in a given frame
Suppose that I have a dictionary where keys are categorical columns and values are their frequencies. How can one replace each category with its frequency on the entire frame
Example with pandas data frame
fe = h2oDF_fe.groupBy(col).size()/len(fe)
h2oDF_fe.loc[:, col_fe] = h2oDF_fe[col].map(fe)Is there for h2o frames an equivalent function to pandas dataframe's map function
10 replies
On my ubuntu laptop, when xgboost is included in AutoML, it is crashing.
If I run just
xgboost(..., backend = "auto") it is also crashing.However,
xgboost(..., backend = "cpu") works.I've tried with several h2o versions.
I could provide a simple reprex and session info.
2 replies
with continuous features.
1 reply
Hi
With a dictionary (imputer_dict) where keys are the same as my frame's columns I wanted to fill NAs in a frame with that dictionary's values as it is possible with pandas dataframes (e.g DataFrame.fillna(value = imputer_dict))
For each column (x) in my frame I wanted to replace its NAs with its value in imputer_dict (i.e imputer_dict[x])
If anybody have an idea
Thanks
12 replies
Hi
I was trying to impute a frame with dictionary (each feature with it’s imputation value) :frame.impute(values=[imputer_dict[col] for col in frame.columns])
I wanted to skip some columns (e.g x) by adding None as value in imputer_dict as said in the documentation and I’m guetting th following error
The documenation is confused to me. Anybody has an idea of what caused the error
2 replies
Hi,
I have a question.
A quick background on what I did with H2O is that I am able to perform predictions in my own Java environment using the MOJO that I downloaded from H2O.
I am planning to use the outputs (Scoring History, Variable Importances, etc) from the model that I built in H2O.
I want to use this data and display them in my own development environment.
I have checked H2O's documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-genmodel/javadoc/index.html).
However, I can't seem to find any sample that shows this.
I want to ask. Is it possible to retrieve and display this data using the MOJO?
If so, how can I do this? Is there a documentation reference I can look into for this?
Regards,
Gabriel
Hi,
I have a question about the documentation for the stopping_rounds parameter for h2o’s models: https://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/algo-params/stopping_rounds.html
It says: To disable this feature, specify 0. When disabled, the metric is computed on the validation data (if provided); otherwise, training data is used.
I was wondering if this was a typo; was it meant to say when enabled rather than when disabled? Or does this mean that if enabled, early stopping will be computed using the training data rather than the validation data?
Thanks!
1 reply
Hi. I am running H2O with Python.
So far, what I am able to do is build a GBM model and print its data. Below is my sample code.
gbm_model = H2OGradientBoostingEstimator(ntrees=100, max_depth=4, learn_rate=0.1)
gbm_model.train(predictors, response, training_frame=trainingFrame)
print(gbm_model)
Printing gbm_model displays tables of data like Scoring History and Variable Importances.
What I want to achieve is retrieve each data (with its header name) so that I can map and display those data on my own way.
So, I tried to access the Variable Importances data by looping through it.
print("Loop through Variable Importance Items")
varImp = gbm_model.varimp()
for varImpItem in varImp:
for item in varImpItem:
print(item)
print(" ")
For additional info, gbm_model.varimp() returns a ModelBase object.
However, what was retrieved was only the data itself.
The header names (variable, relative_importance, scaled_importance, percentage) were not included for the display.
I want to ask, is there a way to retrieve the header names for this? If so, how can I do it?
You can retrieve each data from this command
gbm_model._model_json['output’]- For varibale importance:
gbm_model._model_json['output’]['variable_importances’]
And if you want it in a dataframe
var_imp = gbm_model._model_json[‘output’][‘variable_importances'].as_data_frame()
I am using EasyPredicModelWrapper to run trained Models.
For numerical and categorical input values this works fine.
But I noticed that there are problems with time columns: Time is not converted automatically, as in the flow interface, but an error is thrown.
Do Time Columns need to be addressed in a special way?
Regards
Hi,
I was comparing h2o.xgboost vs the native xgboost by following the instructions written in estimator_base.py:
h2o.init()
training_hf = h2o.import_file("train.csv")
h2o_booster = H2OXGBoostEstimator(distribution="bernoulli",
seed=0,
ntrees=10,
max_depth=5,
min_split_improvement=0.1,
learn_rate=0.1,
sample_rate=0.9,
col_sample_rate_per_tree=0.9,
min_rows=2
)
label = "response"
features = training_hf.columns
features.remove(label)
training_hf[label] = training_hf[label].asfactor()
h2o_booster.train(x=features, y=label, training_frame=training_hf)
h2oPredict = h2o_booster.predict(training_hf).as_data_frame()['p1'].values
nativeDMatrix = training_hf.convert_H2OFrame_2_DMatrix(features, label, h2o_booster)
nativeDMatrix.feature_names = features
nativeParams = h2o_booster.convert_H2OXGBoostParams_2_XGBoostParams()
nativeModel = xgb.train(params=nativeParams[0], dtrain=nativeDMatrix, num_boost_round=nativeParams[1])
nativePredict = nativeModel.predict(data=nativeDMatrix, ntree_limit=nativeParams[1])Apparently the predictions are just very close(not because of rounding), but definitely not exactly the same. Did I miss anything from the instruction?
Moreover, the tree structure starts to diverge a lot after a few initial ones being very similar.
2 replies
H2O failed to start, stopping execution.
Hi All,
Has anyone managed to get XGBoost in h2o-3 to use a GPU backend when running in a docker container? And, if so, can you give me some pointers?
I'm running out of ideas and the only output I get from h2o to debug is:
ERRR on field: _backend: GPU backend (gpu_id: 0) is not functional. Check CUDA_PATH and/or GPU installation.
Is there any way to get something more verbose?
For context. I'm using Metaflow in combination with AWS Batch.
- The AWS batch AMI is a slightly customised build on top of the ECS GPU Optimised AMI which includes nvidia driver 418.87.00
- The container image is built on nvidia/cuda runtime centos 7. I've tried every version of CUDA, but currently have cuda 8 and then install the the cuda 9 libraries
- /usr/local/cuda symlink points to cuda 9
- CUDA_PATH and LD_LIBRARY_PATH are set
- h2o version is 3.30.1.1
- Packages and Python 3 version are all managed with conda environment
- Instance type is p3.2xlarge with nvidia Tesla
I have tested whether the container can see the GPU using pynvml and it seems to work. I also ran a test script using tensorflow-gpu and that seemed to work too. That leads me to conclude the problem is with h2o and/or cuda.
The documentation on configuring h2o to use GPUs with XGBoost is pretty limited in scope and as far as I can tell, I'm meeting the requirements.
Any help/advice much appreciated...
Thanks
9 replies
Hi all! Found a bug in error handling:
- h2o\model\model_base.py
- Line 378:
raise ValueError("'test_data' must be of type H2OFrame. Got: " + type(test_data))
type in Python 3.7.4 is not a string, can't concatenate
It should probably be replaced withraise ValueError("'test_data' must be of type H2OFrame. Got: " + type(test_data).__name__)
1 reply
h20 version: 3.30.0.6. it worked somedays before same code is not working now please help currently its working in local only
7 replies
7 replies
Would anyone be able to give me some pointers on optimising resource allocation for XGBoost training in h2o? I'm sure I read in the documentation that you needed to leave a proportion of the available cpu/memory for XGBoost? Is this correct, or should I be giving H2O as much as possible?
For example. If I have an instance with 10 cpu and 100GB RAM what should I allocate directly to h2o and what, if any, should I keep free for XGBoost?
2 replies
I can successfully retrieve a model using h2o.get_model(model_id).
However, I am manually inputting or hard coding the model_id.
Other than H2O Flow (Models >> List All Models), I want to know if there is a way to list all the model IDs in the cluster. Thanks.
2 replies
h2o.explain() features in h2o python.Basically the feature is non-existent in 3.30.1.3 - which is the latest version for python ?
I created an issue here https://h2oai.atlassian.net/jira/software/c/projects/PUBDEV/issues/PUBDEV-7850?jql=project%20%3D%20%22PUBDEV%22%20ORDER%20BY%20created%20DESC
1 reply
FREE value mean? And why am I OOM if I've got 10x as much FREE as K/V and POJO?
When I run grid search with parallelism set to 0 or some value different than 1 it always hangs. In other words, after some time the progress bar can reach 100% but the grid never finishes. It happens with both the Cartesian and RandomDiscrete strategies. The model that I'm using is GBM and the cross validation folds are specified through fold_column.
When I run the grid with the default value for parallelism, some parameter combinations fail due to the dataset being very small (<100 rows) but the grid finishes.
Did anyone have a similar problem? I don't know how to debug this issue.
4 replies
Hi
I'm training a GBM multi classifier and I wanted to know, what could cause the following error. Thanks
raw_df = h2o.import_file()
df = h2o.deep_copy(raw_df[raw_df['x'] > 10, : ], 'df')
df['split'] = df['y'] .stratified_split(test_frac=0.2, seed=1)
train_valid = df[df['split'] == 'train', :].drop('split')
test = df[df['split'] == 'test', :].drop('split')
train_valid['col_split'] = train_valid['y'] .stratified_split(test_frac=0.2, seed=1)
train = df[df['split'] == 'train', :].drop('col_split')
valid = df[df['split'] == 'test', :].drop('col_split')
raw_df['y'].unique().nrow => 95
df['y'].unique().nrow => 93
train['y'].unique().nrow => 93training GBM alog with class_sampling_factors = [w1, ...., W93]
OSError: Job with key $03017f00000132d4ffffffff$_af9c11386cb765249816853dfc3d47fe failed with an exception: java.lang.IllegalArgumentException: class_sampling_factors must have 95 elements
stacktrace:
java.lang.IllegalArgumentException: class_sampling_factors must have 95 elements
at hex.tree.SharedTree$Driver.computeImpl(SharedTree.java:244)
at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:238)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1563)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)or the following one, when using "balance_classes": True in GBM model
OSError: Job with key $03017f00000132d4ffffffff$_acb90549c4fb00eefd9be1d55ab5448b failed with an exception: java.lang.IllegalArgumentException: Error during sampling - too few points?
stacktrace:
java.lang.IllegalArgumentException: Error during sampling - too few points?
at water.util.MRUtils.sampleFrameStratified(MRUtils.java:309)
at hex.tree.SharedTree$Driver.computeImpl(SharedTree.java:252)
at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:238)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1563)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)