Thanks for all your contributions, we’ve just released v2.6.0. Please check out the highlights and release note at https://github.com/optuna/optuna/releases/tag/v2.6.0 or via the Tweet https://twitter.com/OptunaAutoML/status/1368818695250669570. In short, it contains
- Warm starting CMA-ES and sep-CMA-ES support
- PyTorch Distributed Data Parallel support
- RDB storage and heartbeat improvement
- Pre-defined search space with ask-and-tell interface
Howdy!, My name is Eduardo, and I am using optuna for an optimization problem. I am using a search space, and I would like to return more than one metric from the objective. When I try to do so, like in the example below, I get the following error message: "Trial 2 failed, because the number of the values 3 does not match the number of the objectives 1."
def objective(trial):
'''
Instantiate model and evaluate
'''
Metric 1 =
Metric 2 =
'''
Get additional metrics
'''
Metric 3
return [Metric 1, Metric 2, Metric 3]
d_space = np.linspace(0.05, 0.95, 2)
l_space = np.linspace(0.0001, .0002 , 2)
search_space = {"D": d_space, "L": l_space}
study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space))
study.optimize(objective,
n_trials=d_space.shape[0] * l_space.shape[0],
show_progress_bar=True)
3 replies
Hi everyone, I'm a Rookie at optuna and I would like to know how I can change the evaluation metrics for finding the most fitting hyperparameters. If I understood it corectly, the metric for on which optuna decide if a hyperparameter set is a good one is the accurency. Instead of the accurency I would prefere to use the F1-Score or the Recall.
How could that be implemented with optuna?
How could that be implemented with optuna?
2 replies
I'm running into an issue trying to optimize the hyperparameters for a TF Estimator model (specifically a DNNClassifier). When I set up and run an Optuna study it quickly uses up all of my session's resources and crashes (this is using either a high-memory GPU Colab session or an AWS Deep Learning AMI). I haven't had this problem using non-Estimator TF models, nor does it occur when I run my model outside of Optuna, so I'm wondering if there is something special that needs to be done with them.
17 replies
Hi :-)
Is there predefined way to nest trial parameters?
I would like to pass a trial object into a function and all the parameters that get added inside this function should be prefixed automatically by a string that I specify.
I imagine the interface to look something like this but did not find something similar in the API:
def configure_subsytem_a(trial: optuna.Trial) -> SubSystemA:
n_params = trial.suggest_int("n_params", 1,3)
return SubSystemA(n_params)
trial = ...
subsystem_a = configure_subsytem_a(trial.withPrefix('subsystem_a'))This should result in a conifg like this:
{
'subsystem_a.n_parmas': 3
}It would be quite easy to build this functionality myself, by wrapping the trial object, but if functionality like this is provided, I would prefer to use that.
1 reply
Hi,
What is the preferred way of dealing with trial/suggestions that have ranges that depend on each other, see below callfunction...
is the preferred way to do as below or is it better to set like a central value and penalize values outside of range?
Does this method even work with optuna, sampler being used: TPE. Are certain samplers better at this?
Tips on literature?
class Objectiveoptim(object):
def init(self, idmodelsdict, value):
self.idmodelsdict = idmodelsdict
self.value = value
def call(self, trial):
totalfactor = 0
totalvalueused = 0
valuedict = dict()
for id_, model in self.idmodelsdict.items():
model.eval()
valuedict[id_] = trial.suggest_float(id_, 0, self.value - totalvalueused)
totalvalueused += valuedict[id_]
totalfactor += model(torch.tensor([valuedict[id_]], dtype=torch.float32))
return totalfactor
Hi,
I'm having difficulties in understanding how i can use the command
optuna.visualization.plot_param_importances(study)to visualize hyperparameters importance while performing multiple objectives optimization. I undestand that I should specify the metric wrt which I want the importances to be computed but I don't understand how to do so.
Thanks in advance!
7 replies
Hey everyone. I've been using Optuna-dashboard for a couple of weeks now and I'm detecting a weird behaviour. I'm using Optuna 2.6 to optimize the hyperparameters of a relatively small (5 to 8 layers) tensorflow/keras convolution neural network in a Jupyter notebook and optuna-dashboard 0.3.1 (SQLAlchemy 1.3.22) to monitor the evolution of the optimization. My default browser is Chrome (version 89.0.4389.82) and my OS is Windows 10. The strange behaviour that I've started noticing is a very high RAM consumption by optuna-dashboard after a certain number of trial in my optimization studies, much larger than the database file being created by Optuna. For example I have a study.db file that has roughly 11Mb corresponding to 1563 trials points. Displaying this on the browser gobbles up around 4 Gb of RAM and this is if shutdown optuna-dashboard and reload the study.db from scratch. When I continuously monitor the optimization experiment from the beginning, optuna-dashboard reaches around 6 Gb of RAM (for the exactly same file). Around trial 500 (more or less), the browser starts to get unresponsive or very low in terms of selection buttons, etc. This difficults the results analysis and is quite annoying... I report this same behaviour in two different PCs with different graphic cards and memory configurations. My CPUs and GPUs are running always below 60% performance, so lack of resources do not seem to be the cause. Is this suppose to happen? What can I do make the process run faster?
1 reply
import optuna
def objective(trial):
x = trial.suggest_float("x", 0, 5)
y = trial.suggest_float("y", 0, 3)
v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2
v=v0+v1
return vIs it possible to log the value of
v0 and v1 in some functions such asstudy.trials_dataframe. As I also wish to analyse the intermediate values from the objective function.
import optuna def objective(trial): x = trial.suggest_float("x", 0, 5) y = trial.suggest_float("y", 0, 3) v0 = 4 * x ** 2 + 4 * y ** 2 v1 = (x - 5) ** 2 + (y - 5) ** 2 v=v0+v1 return vHello!
Is it possible to log the value ofv0andv1in some functions such asstudy.trials_dataframe. As I also wish to analyse the intermediate values from the objective function.
Solution is given by optuna author here: https://github.com/optuna/optuna/issues/2520#issuecomment-806578752
In the Optuna documentation, there is a note in the "Key Features" section called "Which Sampler and Pruner Should be Used?" and that points to a possible very relevant document " Ozaki et al, Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020" that describes the performance of samplers/pruners pairs for deep learning tasks. Unfortunately this document is in Japanese! Is anyone aware of a translation of this document to English or of some blog or webpage where its content is described in English? Since deep learning tasks are a very contemporary subject, it would be very useful to have this benchmark (or a summary of it) in English so researchers from different places can tap into that information.
1 reply
Thanks to your contributions, we’ve just released v2.7.0. Check out the highlights and release notes at https://github.com/optuna/optuna/releases/tag/v2.7.0 or with the Tweet https://twitter.com/OptunaAutoML/status/1378915791614058502?s=20. Highlights:
Optuna dashboard now has its own repository. Install with
Deprecate
Lots of new tutorials and examples!
Optuna dashboard now has its own repository. Install with
pip install optuna-dashboard and try it out with optuna-dashboard $STORAGE_URL. The dashboard subcommand is now deprecated!Deprecate
n_jobs in Study.optimize keeping process-level parallelization. There should be one obvious way to do distributed optimization.Lots of new tutorials and examples!
I'm in doubt specifically regarding the difference of folds and nfold of LightGBMTunerCV.
Hi all. In a given optimisation problem, I'm not only interested in the solutions, but getting as many non-duplicate solutions for a certain problem. So far, I have been using NSGAIISampler with multiple objectives, but I would like to encourage the evolution (or sample, if I use TPE instead) to "explore" more. Any advice? Thanks
3 replies
I have a very dumb question about Optuna. When I run a study and get study.best_trial.value and my objective function is returning the accuracy of the test / validation dataset, what is that best_trial.value? Is that the accuracy on the test / validation set, or is it the accuracy on the training set? Or is it some other number all together? I just can't seem to find or decipher the answer. Thank you.
2 replies
To be specific, tune params for community detection on graphs.
1 reply
Hi everyone. I want to use optuna for a simple multi-objective optimization problem. It works well, the only problem is that the process of retrieving of best trials takes too much time, even much more than the optimization itself. It's not clear for me why this simple python assignment should take so much time. This is a piece of code to do that, in which I have a function in two dimensions (x, y) and I want to find the value of y that minimizes the function for a given x.
def multi_objective(trial, X):
y = trial.suggest_float("y", -5., 5.)
result = []
for x in X:
result.append((x**2+y-11)**2 + (x+y**2-7)**2)
return result
optuna.logging.set_verbosity(optuna.logging.WARNING)
Y = np.linspace(start=-5., stop=5., num=500)
study = optuna.create_study(directions=['minimize']*len(Y))
study.optimize(lambda trial: multi_objective(trial, Y), n_trials=1000)
# the next line takes too much time to run
bests = study.best_trials
best_values = [trial.values for trial in bests]
best_params = [trial.params for trial in bests]
Hi there! I have some hyperparameters that are dependent on others. For example, I'm testing two different learning rate schedulers in AllenNLP: slanted_triangular and linear_with_warmup, and they each have their own hyperparameters. How can I tie these related hyperparameters to values that are related to each other? Also, is there any way to improve the sampler configuration to take these dependencies into consideration, to prevent combining and evaluating unnecessary scenarios?
6 replies
Like above i have hyperparameters that are dependent on each other. Especially the last hyperparameter in the loop is dependent on all other hyperparameters, will the below code example be a problem? chosen sampler tpe(multivariate)
class Randomoptim(object):
def init(self, modelsdict):
self.modelsdict = modelsdict
self.value = 1.0
class Randomoptim(object):
def init(self, modelsdict):
self.modelsdict = modelsdict
self.value = 1.0
def __call__(self, trial):
totalfactor = 0
valuedict = dict()
for id_, model in self.modelsdict.items():
model.eval()
if id_ == list(self.modelsdict.keys())[-1]:
valuedict[id_] = trial.suggest_float(id_,
self.value,
self.value)
else:
valuedict[id_] = trial.suggest_float(id_,
0,
self.value)
self.value -= valuedict[id_]
totalfactor += model(valuedict[id_])
return totalfactor
Only manually in database?
Well, discovered, that you shouldn't do it, because of corruption for sampler history)
I'm running a study with AllenNLP optuna testing 9 hyperparameters, with 8 categoricals and a float from 1 to 5 (with step 1). This would yield me around 576k different combinations. I'm around trial 26 now, and for some reason, trials from 21 to 25 were all using the exact same hyperparameters, which were "copied" from trial 11, which is the best one. Whis is this happening? Why isn't optuna trying other combinations? I'm using the default TPESampler (also default parameters) and SuccessiveHalvingPruner with min_resource 5 (other parameters are also default). I'm also running two processes, one for each GPU, but both sharing the same optuna database.
5 replies
Hello, Im trying to do some regressions with optuna and after the 50 trials, i want to keep the best r2_score and a scatter plot of "actual vs predicted values". If I do the process inside the objective trial it will return 50 scatter plots, but i only want the best one. Is there any way you can do this?
4 replies
Hi, I am running the following code and I get an error message, could you please help?
import optuna.integration.lightgbm as lgbm
best_params, tuning_history = dict(), list()
booster = lgbm.train(params, dtrain, valid_sets=dval,
verbose_eval=0,
best_params=best_params,
tuning_history=tuning_history)
The error message is:
TypeError Traceback (most recent call last)
<ipython-input-28-c0d324367a3c> in <module>
4 verbose_eval=0,
5 best_params=best_params,
----> 6 tuning_history=tuning_history)
7
~\Anaconda3\envs\tf2\lib\site-packages\optuna\integration_lightgbm_tunerinit.py in train(args, kwargs)
32 _imports.check()
33
---> 34 auto_booster = LightGBMTuner(args, kwargs)
35 auto_booster.run()
36 return auto_booster.get_best_booster()
TypeError: init() got an unexpected keyword argument 'best_params'
Thank you.
1 reply
As always, thanks for all the feedback and contributions. We’ve just released v2.8.0 with several interesting features and improvements.
🙊Constant Liar (CL) for TPE improves distributed search
🌳Tree-structured search space support for multivariate TPE
🪞Copying Studies across storages
📞Callbacks to re-run a pre-empted trial
Check out the highlights and release notes at https://github.com/optuna/optuna/releases/tag/v2.8.0 or with the Tweet https://twitter.com/OptunaAutoML/status/1401799603154939908.
Hi all. I have an objective function that has a lot of invalid regions, which I inform the study by returning a nan. I notice that the TPE spends a lot of time/trials around invalid regions after a while. Is this by construction due to the "uncertainty driven" GP logic? Is there a way of preventing the TPE to spend so long trying points with high uncertainty due to previous nan in the vicinity? Cheers
4 replies
Hello,
I have a question about hyperparameter importance. When I use optuna.importance.get_param_importances(study) and optuna.visualization.plot_param_importances(study) , both on the same single-objective study and with default parameters, the results differ. Why it is like that? or maybe I am doing sht wrong? I tried to look for an explanantion in optuna doc but haven't found anything.
I have a question about hyperparameter importance. When I use optuna.importance.get_param_importances(study) and optuna.visualization.plot_param_importances(study) , both on the same single-objective study and with default parameters, the results differ. Why it is like that? or maybe I am doing sht wrong? I tried to look for an explanantion in optuna doc but haven't found anything.
Hello there optuners, this is my first time using optuna and it works well except for one small thing. I am not exactly sure how to explain this. So I use Weights and Biases website (wandb.com) to track all of the model runs--the logs. And I am using pytorch-lightning to run my model. Now I am successfully able to create the
objective() function and run the training, but it seems that the log on Weights and Biases is getting overwritten. So instead of different runs or experiments for each trial, I just see one really funky trial with training error and losses everywhere--because everything is mixed together.
Here is the code if it helps. I am not sure if there is a way to tell optuna to initiate a new experiment each time the
trial runs. def objective(trial: optuna.trial.Trial) -> float:
input_window_size = trial.suggest_categorical("input_window_size", [20, 30, 40])
output_window_size = trial.suggest_categorical("output_window_size", [ 20, 30, 40])
test_percentage = 0.20
val_percentage = 0.20
lags = [1, 2, 3, 365, 366, 367]
lag_combos = list(powerset(lags))
laglist = trial.suggest_categorical("lags", lag_combos)
drop_columns = ['pr', 'tmmx']
drop_column_combos = list(powerset(drop_columns))
remove_columns = trial.suggest_categorical("remove_columns", drop_column_combos)
batch_size = trial.suggest_categorical("batch_size",[16, 32, 64, 128])
datamodule = get_dataset('Sensor1')
datamodule = datamodule(input_window_size=input_window_size,
output_window_size=output_window_size,
test_percentage=test_percentage,
val_percentage=val_percentage,
laglist=laglist,
remove_columns=remove_columns,
batch_size=batch_size)
datamodule.prepare_data()
datamodule.setup()
hidden_dim = trial.suggest_categorical("hidden_dim", [16, 32, 64, 128])
num_layers = trial.suggest_int("num_layers", 1, 3)
dropout = trial.suggest_float("dropout", 0.2, 0.5, step=0.1)
optimizer_name = trial.suggest_categorical("optimizer", ["Adam", "RMSprop", "SGD"])
lr = trial.suggest_uniform("lr", 1e-5, 1e-1)
model = LitDivSensor(num_features = datamodule.num_features,
hidden_dim = hidden_dim,
num_layers = num_layers,
dropout = dropout,
debug = False,
learning_rate = lr,
batch_size = batch_size,
optimizer_name=optimizer_name)
tb_logger = pl_loggers.TensorBoardLogger('logs/', name='division-bell-rnn')
wandb_logger = pl_loggers.WandbLogger(name='division-bell-rnn'+ str(date_time),
save_dir='logs/',
project='division-bell',
entity='****',
offline=False)
trainer = pl.Trainer(
logger=[tb_logger, wandb_logger],
checkpoint_callback=False,
max_epochs=EPOCHS,
gpus=1 if torch.cuda.is_available() else None,
callbacks=[PyTorchLightningPruningCallback(trial, monitor="val_loss")])
hyperparameters = dict(input_window_size=input_window_size,
output_window_size=output_window_size,
laglist=laglist,
remove_columns=remove_columns,
batch_size = batch_size,
hidden_dim=hidden_dim,
num_layers=num_layers,
dropout=dropout,
optimizer_name=optimizer_name,
learning_rate=lr)
trainer.logger.log_hyperparams(hyperparameters)
trainer.fit(model, datamodule=datamodule)
return trainer.callback_metrics["val_loss"].item()
alternatively: What exactly is the issue with sqlite and running optuna in parallel? I'm planning on running at most 4 instances of optuna in parallel (the machine has 4 GPUs). Is accessing sqlite simultaneously merely not as efficient or will it introduce inconsistencies? I'd be completely OK with, e.g., locking happening, that's still plenty fast for my use case obviously. I'm also on a system that isn't maintained by myself, so I can't easily install postgresql or some other RDB.
For that reason I was also looking into portable versions of Postgresql that would be self contained in a single directory but only found something for Windows (and this is a ubuntu machine I'm talking about)
fwiw I just realised that in https://optuna.readthedocs.io/en/stable/faq.html#how-can-i-use-two-gpus-for-evaluating-two-trials-simultaneously the code suggests that it is OK to use sqlite from multiple instances (on the same machine). Is this viable or just an oversimplification for the sake of easy to read documentation?
I have one question, I have some clusters with limited time for execution, how do I resume a trial ?