optuna/optuna

Main room

James Y

@yuanjames

Anyway, I am testing the idea in a simple way, I am not using suggest function, instead of I define another method to generate parameters. sampe_relative and sample_independent, infer_relative_search_space three function defined with 'pass'.

Hiroyuki Vincent Yamazaki

@hvy

Ah okay, you were speaking from a sampler’s perspective. Generally speaking, if you want to implement a sampler, you must take into account that the search space may or may not change between trials. For your particular objective function, you might be able to make stricter assertions.

Are you changing the internals of the trial/study classes, while overriding those methods with pass? In any way, it sounds a bit tricky or unusual so let us know if you have further questions.

James Y

@yuanjames

@hvy Hi, thanks so much for your support, my code is still based on optuna framework, but I skipped the samper (for saving time, test idea quickly). I directly wirte the sampling method in objective function before defining model. So that, the parameters are not longer sampled from sampling relative_().

James Y

@yuanjames

class Sampler():
def init(self):
pass

def infer_relative_search_space(self, study, trial):
    return
def sample_relative(self, study, trial, search_space):
    return
def sample_independent(self, study, trial, param_name, param_distribution):

    return

def objective(trial):

paramters = self_defined_sampling()
y = model(x, paramters)

if name == "main":
optuna.create_study(objective, Sampler = Sampler() )
study.optimise()

In this case, I still can get part of training information benefited by Optuna

Andrew Annex

@AndrewAnnex

hey all, although I should probably use gridsearch/a for loop for this, but I have a categorical parameter with 16 possible values and when I use optuna with suggest_categorical it doesn't do an exhaustive search (or it repeats suggestions), should I just not use optuna for this (I will be tuning other hps later though so it is neat to use 1 api)

James Y

@yuanjames

SimulatedAnnealing example reports an error, Exception has occurred: ImportError
cannot import name 'TrialState' from 'optuna.trial'

Should be optuna.struct.StudyDirection and optuna.struct.TrialState,

Andrew Annex

@AndrewAnnex

oh there is a grid sampler

anony-account

@anony-account

@sile Thank you very much, I have found a way to fix it.

Takeru Ohta

@sile

:tada: > @anony-account

@AndrewAnnex Since v1.2.0, Optuna has provided GridSampler (it's still experimental feature though).

Takeru Ohta

@sile

@yuanjames Although it might be somewhat confusing to existing users, the optuna.structs was deprecated and new classes (e.g., optuna.trial.TrialState and optuna.study.StudyDirection ) were introduced by the last release (v1.4.0]. Sorry for the inconvenience, but you need to use the latest Optuna to run the example.

James Y

@yuanjames

Ok thanks

Zishi Wu

@zishiwu123

Hi everyone! My name is Zishi and I discovered Optuna this week after watching your talk in Scipy 2019. I was searching for open-source hyper-parameter optimization libraries for a project requiring regression. Thank you making this project open-source!

I noticed all the pruning examples are performing classification, usually on the MNIST dataset.
https://github.com/optuna/optuna/blob/master/examples/pruning

The only regression example I could find was Keras + MLflow, on the wine quality regression dataset.
https://github.com/optuna/optuna/blob/master/examples/mlflow/keras_mlflow.py

I combined code from the keras_mlflow.py example with the keras_integration.py example in the pruning folder to get a Keras + Regression + Pruning script
that evaluates Mean Squared Error on the wine quality regression dataset. I wanted to ask if you would like to my regression script in your examples/pruning folder? I know it's just a copy of two other examples you already have but I think the pruning examples could use at least one regression example. Please let me know what you think.

Also does anyone here have experience in using Optuna for tuning regression models like XGBoostRegressor or Keras Sequential? I am wondering if there is links to guidelines on which parameters to tune first for a regression tasks. I would very much appreciate any experience as I understand there is a lot of trial and error that goes into model tuning.

Zishi Wu

@zishiwu123

Finally, I have a question about hyper-parameter optimization. If I run a study with 100 trials, and suppose it gives me a best hyper-parameter configuration for XGBoost for that study. Then I run another study with 100 trials, and it will give me a different best hyper-parameter configuration for XGBoost for that study. And so on and so forth. How do I choose what hyper-parameter configuration then? Since we are randomly sampling a search space, then the samples for each study will be different correct?
I thought maybe it doesn't matter if we get different "best" hyper-parameter configurations for different studies. As long as the performance on the evaluation metric is good. So study 1, 2, and 3 may give me hyperparameter configurations 1, 2, and 3 that are somewhat different. But as long as they don't have high variance in the evaluation metric (e.g. 95%, 93%, 96% for accuracy), then the Optuna hyper-parameter optimization library has performed well. However, if there is a large variance in the performance of the models (e.g. 81%, 94%, 73%) that result from the hyper-parameter tuning process of Optuna, then we have a problem. We would then have to shift to a different algorithm? Maybe shift from XGBoost to LightGBM or some other algorithm and see if that algorithm gets more stable results with Optuna for this specific dataset we are working on.
I am new to hyper-parameter optimization and would appreciate any comments and feedback. Especially because I think many of you are Data Scientists and Machine Learning Engineers, while I am still a grad student. I would like to pick your brain on how people deal with these issues in the "real world".

Kev. Noel

@arita37

@hvy :

Have posted an article on meta-learning meta-dataset, think there is a strong relation with hyper-parameters and meta-learning

Hiroyuki Vincent Yamazaki

@hvy

@arita37 I might have missed something down the line. That sounds intereseting but could you fill me in?

Hiroyuki Vincent Yamazaki

@hvy

@zishiwu123 Nice, and welcome. I personally think it’s okay to add an example if it’s informative in some sense. The cons are increased maintenance costs and perhaps inconsistencies with other examples. As you say, I image most of the code will be duplicated in some sense and am wondering if we shouldn’t instead reference keras_mlflow.py from keras_integration.py for those who are looking for a regression example, vice-versa. I’m not sure this would’ve helped you address your problem but please let me know what you think.

Hiroyuki Vincent Yamazaki

@hvy

I’m definitely not the best person to answer your question about HPs in Keras but generally speaking, the order highly depends on your model and use case. Learning rates and the choice of optimizers could be reasonable HPs to tune first though? As for XGBoost, this blog might give you some inspiration https://medium.com/optuna/lightgbm-tuner-new-optuna-integration-for-hyperparameter-optimization-8b7095e99258. It addresses importances of HPs with LightGBM.

Hiroyuki Vincent Yamazaki

@hvy

There are probably several of aspect regarding your question with the variance. First and foremost, the variance could be high with Optuna if you don’t run a “sufficient” number of trials, i.e. until the HPO converges. Although 100 sounds reasonable, again, this number is problem-dependent and you could for instance use Optuan’s visualization features to study how your HPO behaives https://optuna.readthedocs.io/en/latest/reference/visualization.html. It then of course depends on the optimization algorith. It seems like you’re using random search which if run until convergence shouldn't manifest the issue you described. Optuna also has a TPE sampler as an alternative to random search, which should converge faster, but as for the variance, maybe @mamurai1208_twitter has a more theoretical answer. Note that TPE is the default algorithm in Optuna and you can see a list of different algorithms (samplers) here https://optuna.readthedocs.io/en/latest/reference/samplers.html. And yes trying different models (incl. frameworks) will of course yield different results if the landscape of the HPO changes. I don’t thinks there’s one straight answer to this. It’d be great if you could provide actual results as you’ll probably be helping us and others along the way.

Jakub

@jakubczakon

Hi, all

We've just updated our Neptune integration with Optuna and now the interactive optuna.visualizations can be logged and rendered in the UI:

optuna_utils.log_study(study)

You can also log and update those visualizations after every iteration to get a better picture while it is training:

neptune_callback = opt_utils.NeptuneCallback(log_charts=True)
study.optimize(objective, n_trials=100, callbacks=[neptune_callback])

It looks like this:

I hope this is useful.

Here is a link to docs.

Toshihiko Yanase

@toshihikoyanase

:cool:

Takeru Ohta

@sile

Cool!

Hiroyuki Vincent Yamazaki

@hvy

:cool:

Andrew Annex

@AndrewAnnex

@sile thanks I got it to work

although I ran into an issue with the plotting functions in optuna, it would be great if there was a matplotlib fallback for the plotly plots as I probably have some config issues with my jupyter lab at the moment, not a blocker for me though

Zishi Wu

@zishiwu123

@hvy I think it would be better to include a link to keras_mlflow.py, within the the header documentation for keras_integration.py, and say that link contains a regression example using Keras.

Thank you for the pointers to the different sampler algorithms. I think I will first try the default TPE algorithm with Keras and LightGBM regression models. I will split my data into 80% for training/validation and 20% for testing. Run 10 studies, each containing 100 trials, on 80% of the data and get 10 optimal hyperparameter configs. Then initialize 10 models with those hyperparameter configs, train each on the 80% of training/validation data, and predict the 20% holdout data for testing. Then measure the average and the variance in the mean squared error of those 10 models on the testing set. It will probably take a week for me to get the results for Keras model. I will keep you posted.

Hiroyuki Vincent Yamazaki

@hvy

:thumbsup:

This is just my personal opinion but including a link that way sounds quite reasonable. Would you like to create a PR? By the way, I’m wondering whether we shouldn’t change the dataset used in keras_mlflow.py to e.g. Boston.

Zishi Wu

@zishiwu123

@hvy I would be honored to make a PR to include the link to keras_mlflow.py within keras_integration.py. As for the Boston dataset, I think it would make sense to use it as the standard regression example. The problem with the wine quality dataset is that the target variable is not continuous and only has 3 different values. In comparison, the Boston dataset uses housing price as the target variable, which is continuous.

I can start with a PR to include a link to keras_mlflow.pyin keras_integration.py. Afterwards I can work on recreating the keras_mlflow.py with the Boston dataset instead of wine dataset. How does that sound?

Takeru Ohta

@sile

@AndrewAnnex Thank you for your feedback. If the fallback support is really necessary to you, I'd appreciate it if you could create a feature request issue ( https://github.com/optuna/optuna/issues/new/choose ) with the detailed explanation about the plotly problem.
(I'm not familiar with plotly and matplotlib so I don't know how it's easy to support matplotlib though)

Hiroyuki Vincent Yamazaki

@hvy

@zishiwu123 That sounds good to me. Let us know if you have any concerns creating those PRs. I’m just wondering whether @harupy has some comments regarding the dataset in the regression example as the author of it.

Takeru Ohta

@sile

FYI: I found a note about using visualization functions on JupyterLab (Optuna API Reference -- Visualization) > @AndrewAnnex

Hiroyuki Vincent Yamazaki

@hvy

Thanks for your contributions as always, we’ve just released a minor update v1.5.0! Trying to wrap things up for v2.0, this release doesn’t include as many commits as in the previous one but still comes with several new and interesting (incl. experimental) features. Here is the release note https://github.com/optuna/optuna/releases/tag/v1.5.0 and our Tweet https://twitter.com/OptunaAutoML/status/1267377905815842816 if you’d like to catch up on the details .

Toshihiko Yanase

@toshihikoyanase

:tada:

Hiroyuki Vincent Yamazaki

@hvy

By the way, for those working with PRs, a neat feature for previewing changes to the documentation was introduced quite recently optuna/optuna#1247 .

Zishi Wu

@zishiwu123

@hvy Did @harupy give comments on whether the wine regression dataset example in keras_mlflow.py should be replaced with the Boston dataset?

Zishi Wu

@zishiwu123

Also great job on version 1.5.0 of Optuna! TheLightGBMTunerCV class is amazing. It saved me a lot of time today. Using it with default settings, I consistently got lightgbm models that had 30% less prediction error than my baseline prediction values on my regression dataset of 10k samples. And it only takes a minute to run the study. Whereas each cross validated Keras trial took 1 minute and a study of 100 trials took 1 - 2 hours.

Hiroyuki Vincent Yamazaki

@hvy

Not AFAIK. If others are fine with it, then there’s no need to change it actually. Open for discussion.

Thanks for sharing your use case, that’s great. It seems like the CV component was one of the most appreciated additions in this release. There was a blog being created a couple of minutes or hours after the release about it.

Pariente Manuel

@mpariente

Hi Optuna people, thanks for the tool, it looks great!
How would I go about optimizing the same objective on different machines, with different jobs?
I didn't find examples in the docs about this

Philip May

@PhilipMay

@mpariente read this: https://optuna.readthedocs.io/en/latest/tutorial/rdb.html

You need an SQL database and use that to coordinate the different optuna machines for the same objective

Pariente Manuel

@mpariente

Thanks ! I'll try today !

Pariente Manuel

@mpariente

So this works in my simple example (foo.py is the example from the CLI doc). I launch the studies asynchronously and sync the DB.

study_name=first_study
STUDY_NAME=`optuna create-study --storage sqlite:///example.db --study-name $study_name --skip-if-exists`
for i in $(seq 1 100); do
    # Replace by job submission here
    optuna study optimize foo.py objective --n-trials=1 --storage sqlite:///example.db --study-name $STUDY_NAME &
done

Are there reasons not to do that? Conflicts in DB or anything I should be worried about?
PS: I have no knowledge about DBs and SQL, yet

Where communities thrive

People

Repo info

Activity