eXtreme Gradient Boosting (GBDT, GBRT or GBM) Library for large-scale and distributed machine learning, on single node, hadoop yarn and more.
I am trying to compile XGBoost and use the python-package in my Mac but cannot get rid of the
XGBoostLibraryNotFound: Cannot find XGBoost Libarary in the candicate path, did you install compilers and run build.sh in root path? error
Been following the tips from https://github.com/dmlc/xgboost/blob/master/python-package/build_trouble_shooting.md page but can get it work
I recently reported a bug dmlc/xgboost#1995 and provided a fix with PR dmlc/xgboost#1996
was wondering if there's anything else to do for a contribution (like writing to a mailing list etc)?
I am working on a big data analytics project, which has frequent updates. I have to perform a lot of analytical queries. Can u guys suggest me the tech stack, I should use for the project?
I am thinking HBase and Hadoop but since I'm new to big data world, I'm kind of confused. Thanks in advance.
I am thinking HBase and Hadoop but since I'm new to big data world, I'm kind of confused. Thanks in advance.
How many columns Terabytes of data? If it is small < 1TB (max up to 5TB) probably a single machine could hold it all in memory (for very small $$). If you consider your data too big. (But still think it all fits into memory i.e. fast queries) maybe https://www.snappydata.io is good. For medium sized data Presto and Impala will be interesting and for very big data Hive. What would you expect for query latency? Certainly spark (unless you are using petabytes of data) would be an intersting fit as well and with something like snappy probably sort usable for fast one off queries.
but since I'm more familiar with SQL, can I do something transactional with low latency in hadoop cluster?
yes, I just have GBs of data
Does that mix well?
If it is only a coulbe of 100reds of GB you really should think if hadoop is the right solution. Anyway regarding SQL: do you need transactions or not? If you just want to output the results of analytical queries probably not. Check https://hbase.apache.org/acid-semantics.html for HBase in case transactions are mandatory. Mabe https://phoenix.apache.org is a fit for your needs.
I think I'll go with phoenix and hbase. thank you so much.
you're so kind and intelligent
hey, been trying (and failing) to implement distributed XGBoost4j using Rabit (like you guys did for Spark and Flink). Do I just need to start a RabitTracker, pass the
rt.getWorkerEnvs() to all subnodes, start Rabit.init(env), do train and Rabit.shutdown() on each and just call rt.waitFor(0) on the driver, or is there something more to it? Seems my train instances aren’t communicating with eachother and are just training using local data
Hello! I'm sorry if I missed this information somewhere on Google, but what's the best way to remove
xgboost after a global install with sudo python setup.py install? I didn't realize there was a pip installable package already and I'd rather do that in my virtualenv
It looks like v0.6.0 is the last stable release. This is about one year old. I'd like to add an update to the hunter (CMake) package manager (last version was 0.4.0), and am curious if this is the recommended version, if a new release is planned in the near future, or if there is some more recent tested tag/commit that should serve as a stable release point. Thanks!
I checked the code, and it seems that it occurs while preparing histograms
More interestingly, when I set missing to some number, e.g. -9999 for xgboost modeling parameters and pass a dataset that doesn't have missing values, AUC drops to 0.5 from 0.65. This is possibly an unrelated issue, but it seems handling of missing values causes multiple problems
import xgboost gives the warning cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
but it looks like it should be fixed in rhiever/tpot#284
was that never merged?
xgb = XGBRegressor(n_estimators=1000, silent=0)
xgb.fit(train.as_matrix(), trainY, verbose=1, eval_metric="rmse")
P = xgb.predict(test.as_matrix())
it doesn't output RMSE metric during training. Where am I wrong?
i was looking to build xgboost from source
just wanted to know if anyone on the group has tried doing it before?
@ckchow you have probably meant columns removed and yes this is the only solution i see right now. The problem is that i have to construct DMatrix from sparse libsvm file, and for example to perform greedy feature selection i would have to create new (big) libsvm file every iteration. Which is annoying.