I think that depends on your hyperparameter. If it is discrete, suggest_discrete_uniform is usually suitable for it.
Of course, you can manually discretize the return value of suggest_float, but I don't think it is a good approach since the observation (= history of hyperparameters) is different from the values used actually.
I'm currently using
suggest_uniform
but I'm considering using
suggest_discrete_uniform, largely because it's more interpretable
Like, if I have
x=0.98 after optimization, is it actually just converging to 1.0
And I started wondering if this has other performance implications
so, I assume that if you did
suggest_categorical on [0.0, 0.1, 0.2...], it would perform substantially worse, right?
Using
suggest_categorical likely to show worse performance than suggest_discrete_uniform because the samplers cannot see the distance between values.
19 replies
If you view each Optuna study as an optimization process, I can then just nest it in an outer optuna process and perform pruning strategies :)
3 replies
Hi there, has anyone else experienced this issue with CmaEsSampler and the PostgreSQL storage backend:
(psycopg2.errors.StringDataRightTruncation) value too long for type character varying(2048)
[SQL: INSERT INTO trial_system_attributes (trial_id, key, value_json) VALUES (%(trial_id)s, %(key)s, %(value_json)s) RETURNING trial_system_attributes.trial_system_attribute_id]
It seems the serialized version of the CMA class is too large to fit in the 2048 character column. This code is the culprit:
optimizer_str = pickle.dumps(optimizer).hex()
study._storage.set_trial_system_attr(trial._trial_id, "cma:optimizer", optimizer_str)
Hi there, I have a issue with optuna.intergration.lightgbm.
If I ran bellow code, got TypeError.
I looked up similar issue, but I didn't found.
from optuna.integration import lightgbm as lgb
models = []
best_params_list = []
best_params, tuning_history = dict(), list()
for train_index, valid_index in skf.split(X_train_valid.values, y_train_valid.values):
lgb_train = lgb.Dataset(X_train_valid.iloc[train_index, ], y_train_valid.iloc[train_index, ])
lgb_valid = lgb.Dataset(X_train_valid.iloc[valid_index, ], y_train_valid.iloc[valid_index, ], reference=lgb_train)
model_ = lgb.train(
lgbm_params,
lgb_train,
valid_sets=lgb_valid,
num_boost_round=1000,
early_stopping_rounds=100,
verbose_eval=10,
best_params=best_params,
tuning_history=tuning_history,
random_state=RANDOM_STATE
)
models.append(model_)
best_params_list.append(best_params)
tuning_history.append(tuning_history)TypeError Traceback (most recent call last)
<ipython-input-27-d60ed207f14a> in <module>()
16 best_params=best_params,
17 tuning_history=tuning_history,
---> 18 random_state=RANDOM_STATE
19 )
20
/opt/conda/lib/python3.6/site-packages/optuna-1.5.0-py3.6.egg/optuna/integration/_lightgbm_tuner/init.py in train(args, **kwargs)
34 _imports.check()
35
---> 36 auto_booster = LightGBMTuner(args, **kwargs)
37 auto_booster.run()
38 return auto_booster.get_best_booster()
TypeError: init() got an unexpected keyword argument 'best_params'
5 replies
Thanks for your contributions as always, we’ve just released a minor update v2.1.0! This release contains more PRs involving documentation, examples and code fixes than ever before. Here is the release note https://github.com/optuna/optuna/releases/tag/v2.1.0 and our Tweet https://twitter.com/OptunaAutoML/status/1302831704940269568 if you’d like to catch up on the details.
When I do HP search on small datasets I often do a cross validation. With cross validation I train in the same trial with same HP params but different data (folds) for multiple times (number of folds). The result is the average of all folds. Is there an out of the box mechanism in optuna to prune these folds after a certain number of folds?
1 reply
Hey guys! I've posted a question on Stack Overflow asking about HP tuning of a LSTM model which requires CoNLL-U files as input: https://stackoverflow.com/questions/63850904/conll-files-in-hyperparameter-tuning-using-optuna
1 reply
Just a heads up, isort was just introduced. For those sitting on ongoing PRs or local changes, it’s a good idea to sync with the latest
master to check for violations (isort . --check --diff). I temporarily configured the repository to require that PRs are verified against the latest commit in master so they won’t get merged otherwise. Sorry for the additinal work... optuna/optuna#1695
In your github lightgbm_tuner_simple.py demo code, you have this line: prediction = np.rint(model.predict(val_x, num_iteration=model.best_iteration)). Is num_iteration=model.best_iteration really needed? Since the train() has early_stopping_rounds set already, wouldn't the best_iteration be used anyway?
I've been trying to work out how to optimize the hyperparameters of a Bi-LSTM model for PoS and dependency parsing (https://github.com/datquocnguyen/jPTDP). The model takes CoNLL-U files as input and I am clueless as to how I can go about using them with Optuna. Any idea? All helps hugely appreciated!
7 replies
Sorry, branch protection was set to enforce each PR to be tested against the latest
master at all times. This was because isort was introduced and the master could break quite easily in terms of style violations. However, we noticed it puts a lot of burden on the PR author so I disabled this enforcement again. It’s now back to normal
Is there anybody who has faced the following problem: AttributeError: module 'optuna' has no attribute 'create_study'? I cannot find out what I'm missing, I just installed Optuna in two computers, one is working fine, but the other I get this message for the code: import optuna and study = optuna.create_study(). Help!
1 reply
study = optuna.create_study(direction='maximize')
objective = MyObjective(x_train, x_val, y_train, y_val)
study.optimize(objective, n_trials=n_trials, n_jobs=n_jobs)
Hi, I was wondering if one can provide some existing results (e.g. from manual hyperparameter tuning) to a study, so that the suggestions for future trials can account for some "trails" outside of optuna? Say one found a good set of hyperparams by chance, my assumption would be that it's beneficial to the optimisation to be aware of it.
2 replies
Hi, I was wondering if there are any companies using Optuna in production? I can see a list of projects using Optuna here. I am looking for some more examples or a more in-depth explanation of how a company is using Optuna.
2 replies
Hi, I was looking for examples of pytorch-ignite & Optuna and had trouble finding anything rather than:
https://github.com/optuna/optuna/blob/master/examples/pytorch_ignite_simple.py
Do you know some? Thanks (:
https://github.com/optuna/optuna/blob/master/examples/pytorch_ignite_simple.py
Do you know some? Thanks (:
Hi, I have extracted best params for LightGBM regressor using optuna and fed these best params back to lightgbm, but I am getting a very different result when training the model as compared to the value obtained during optuna trials with best params. Can someone help? Thanks very much
gbm.set_params(**study.best_params,random_state=42)
gbm.fit(
X_cv_train,
y_cv_train,
feature_name = feature_set,
categorical_feature = cat_features,
eval_set=[(X_cv_test, y_cv_test)],
verbose = True
)
gbm.set_params(**study.best_params,random_state=42)
gbm.fit(
X_cv_train,
y_cv_train,
feature_name = feature_set,
categorical_feature = cat_features,
eval_set=[(X_cv_test, y_cv_test)],
verbose = True
)
Thanks for all your contributions, we’ve just released v2.2.0! Some PRs include the following.
- An update to the
TPESamplerto do relative sampling. This is an experimental feature but please try it out as it’s showing very promising results on early benchmarks. - For those who are using AllenNLP,
AllenNLPExecutornow supports pruning to speed up your NLP experiments. - Many bug fixes, documentation, examples and code fixes!
This is the release note https://github.com/optuna/optuna/releases/tag/v2.2.0 and our Tweet https://twitter.com/OptunaAutoML/status/1312972774193864704?s=20 if you’d like to catch up on the details.
Also, note that starting from this version, Python 3.5 is no longer officially supported.
A fun read if you want to know a little about the internals of the TPE sampler and the recently introduced
multivariate parameter https://twitter.com/OptunaAutoML/status/1313388536192614400
Hi guys, thanks for the awesome framework!
I am author of Catalyst framework - we have recently introduce full Catalyst Config API + Optuna integration
https://github.com/catalyst-team/catalyst/tree/master/examples/cifar_stages_optuna
generally speaking, we have plans for active AutoML development with Config API and Optuna support and I wonder if you are interested in such collaboration too
if so, feel free to join our slack here https://join.slack.com/t/catalyst-team-core/shared_invite/zt-d9miirnn-z86oKDzFMKlMG4fgFdZafw
I would be happy, if @toshihikoyanase @hvy @HideakiImamura could join :)
I am author of Catalyst framework - we have recently introduce full Catalyst Config API + Optuna integration
https://github.com/catalyst-team/catalyst/tree/master/examples/cifar_stages_optuna
generally speaking, we have plans for active AutoML development with Config API and Optuna support and I wonder if you are interested in such collaboration too
if so, feel free to join our slack here https://join.slack.com/t/catalyst-team-core/shared_invite/zt-d9miirnn-z86oKDzFMKlMG4fgFdZafw
I would be happy, if @toshihikoyanase @hvy @HideakiImamura could join :)
1 reply
Hi all. I've been using optuna and a few others. Some initial thoughts here if it is of interest. I'm surely missing something. https://www.microprediction.com/blog/optimize