Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Philipp Dörfler
    @phdoerfler
    For that reason I was also looking into portable versions of Postgresql that would be self contained in a single directory but only found something for Windows (and this is a ubuntu machine I'm talking about)
    fwiw I just realised that in https://optuna.readthedocs.io/en/stable/faq.html#how-can-i-use-two-gpus-for-evaluating-two-trials-simultaneously the code suggests that it is OK to use sqlite from multiple instances (on the same machine). Is this viable or just an oversimplification for the sake of easy to read documentation?
    1 reply
    Renato Hermoza Aragonés
    @renato145
    Im using postgres while training in kubernetes and slurm cluster
    I have one question, I have some clusters with limited time for execution, how do I resume a trial ?
    2 replies
    Shalini
    @tomshalini
    Hello everyone,
    I want to plot history of trials using "plot_optimization_history". It is not working with multi-objectives. Is there something I can try to visualize multi objective sampler results?
    2 replies
    Evgeny Frolov
    @evfro

    Hi everyone. I have a three-level question about the best practices for skipping trials.

    1. Is it possible to prohibit sampling of certain combinations of values from hyper-parameter distributions? I have a complex search space and there's an interplay between sampled hyper-parameter values. As an example, after params a, b, c are sampled the trial is allowed to run only if, e.g., a * b > c.

    2. What is the most appropriate way to skip a trial before performing main calculations in an objective (e.g., defining model, running training, etc.)?
      Currently, once I made all the necessary trial.suggest_* calls I also call my custom is_valid(trial.params) function, which tells me if the trial is allowed to run or not. Naïve implementation is to just call is_valid function within the objective body. But I would like to detach the objective from verification of trial parameters. I compose many models with different objectives and with different restrictions of the search space, so I'd like to avoid adding is_valid(trial.params) call to all objectives and would prefer a cleaner solution, something like a pre-trial check, so that I only define it once and register it for any objective whatever it is. Does optuna has such functionality? Or do I have to go with closures/decorators/custom objective classes?

    3. After the way to skip a trial is found, what would be the correct approach for actually skipping it in the study? I'm specifically interested in the case of TPESampler, which uses history of trials for its next steps. There seems to be at least 2 ways to do so: either raise optuna.TrialPruned() if trial should be skipped, or generate a custom exception like raise ConfigError and add it to the catch list of study.optimize call. Which one is more appropriate in terms of ensuring the correct work of TPESampler?

    Thanks!

    8 replies
    Khawaja Umair Ul Hassan
    @umairkhawaja
    I modified the pytorch simple code example given in this repo and replaced the training/validation loop with a code similar to the sample training loop given in a classifier training example in the pytorch docs. However, the study ends after a single trial. I'm assuming this has something to do with the return accuracy statement at the end of the objective function. Can someone explain the purpose of this statement, or better if someone has any idea what I could be doing wrong?
    1 reply
    ik9999
    @ik9999
    Hello. I use remote pgsql database as a storage. And it really slow because i add many user attributes one by one. Is there a way to insert them all at once?
    7 replies
    ik9999
    @ik9999
    Another question, is there a sampler which goes through the all sets of parameters starting with the most diverse ones (most different from each other)?
    1 reply
    Hiroyuki Vincent Yamazaki
    @hvy
    We just started a user survey about Optuna, how we're all using it and how it can be improved. Any feedback it really appreciated. We'll try to incorporate those into the coming releases, long and short term. 🙇‍♂️ https://docs.google.com/forms/d/e/1FAIpQLScv7Lz7ckbDxdFYHnnXiBDUfPj-cOp9csNbYLuXg31GTj1cTA/viewform
    A.G.
    @Divide-By-0
    hey is there a way for optuna to print the found sensitivities of hyperparameters so i know what is most important to change in the future?
    3 replies
    A.G.
    @Divide-By-0
    How does PyTorchLightningPruningCallback utilize all the graphs of past train losses (or whatever the monitor variable is set to)? If I set the optuna objective() function to return a different parameter (i.e. validation error), will the monitor argument get confused, or still compare current train loss against past train losses?
    1 reply
    Paulo Lacerda
    @placerda
    Hello, is it possible to re-run a trial that failed because I stopped program execution? When I run optimize it does not try to execute that trial again.
    3 replies
    Miguel Crispim Romao
    @romanovzky
    Hi all. I was wondering if there was a quick way to interrupt a study once I have a number of trials that satisfy a condition. For example, I might set a certain boolean user_attr inside the objective function. After a certain number of n trials fulfil that condition, I want to stop. What would be the recommended way of going about this? I was thinking about accessing the trial.study attribute, as it is stateful of the whole optimisation (is this true?) and then invoque trial.study.stop() method after parsing the trial.study.trials to count the my user_attr values. Would this be a good idea?
    1 reply
    A.G.
    @Divide-By-0
    I accidentally ran a trial with objective = maximize instead of objective = minimize, can I switch it now and load from the same db so it can use the found sensitivities? I get ValueError: Cannot overwrite study direction from [<StudyDirection.MAXIMIZE: 2>] to [<StudyDirection.MINIMIZE: 1>]. when I try
    I.e. can I just overwrite the study_directions table manually to have minimize
    Philip May
    @PhilipMay
    I would suggest to create a new study with right direction and then enqueue all trials from your wrong study to the new.
    A.G.
    @Divide-By-0
    Cool! Will try
    SurajitTest
    @SurajitTest
    Hi, I am using xgboost 1.4.0 and optuna 2.8.0. I am using a RHEL 7.1 machine that has 64 processors. I am trying to use 60(out of 64) processors both for xgboost and optuna(study.optimize). The code did run for 8 hours and there is no output. Can someone pls help
    # Import data into xgb.DMatrix form
    dtrain = xgb.DMatrix(X_train,label=y_train)
    dtest = xgb.DMatrix(X_test,label=y_test)
    #Set parameters
    n_splits=5
    random_state=42
    tree_method='auto'
    predictor='cpu_predictor'
    eval_metric='aucpr'
    grow_policy='depthwise'
    booster='gbtree'
    base_score=0.5 
    objective_ml='binary:logistic'  
    verbosity=1
    early_stopping_rounds=100
    num_boost_round=50000
    parallel_jobs=60
    #Instantiate the stratified folds
    skfolds = StratifiedKFold(n_splits=n_splits,shuffle=True,random_state=random_state)       
    # define the search space and the objecive function
    def objective(trial):
        # Define the search space
        param_sp = {
            'base_score'            : base_score, 
            'booster'               : booster, 
            'colsample_bytree'      : trial.suggest_discrete_uniform('colsample_bytree',0.7,0.85,0.05),
            'learning_rate'         : trial.suggest_loguniform('learning_rate',0.01,0.1),
            'max_depth'             : trial.suggest_int('max_depth', 4,8,1),
            'objective'             : objective_ml, 
            'scale_pos_weight'      : trial.suggest_uniform('scale_pos_weight',1,100),        
            'subsample'             : trial.suggest_discrete_uniform('subsample',0.5,0.85,0.05),      
            'verbosity'             : verbosity, 
            'tree_method'           : tree_method,            
            'predictor'             : predictor, 
            'eval_metric'           : eval_metric,
            'grow_policy'           : grow_policy        
        }
        #Perform Native API cross validation  
        xgb_cv_results=xgb.cv(param_sp,dtrain,stratified=True,folds=skfolds,metrics=eval_metric,
                              num_boost_round=num_boost_round,early_stopping_rounds=early_stopping_rounds,
                              as_pandas=True,verbose_eval=False,seed=random_state,shuffle=True)
        # Set n_estimators as a trial attribute
        trial.set_user_attr("n_estimators", len(xgb_cv_results))
        #Obtain the number of estimators 
        n_estimators=len(xgb_cv_results)
        #Print the params selected in the trial 
        #Create the params set for obtaining the cross validation score 
        params_cv={
        'learning_rate': trial.params['learning_rate'], 
        'subsample': trial.params['subsample'],
        'colsample_bytree': trial.params['colsample_bytree'],
        'max_depth': trial.params['max_depth'],
        'scale_pos_weight': trial.params['scale_pos_weight'],
        'base_score':base_score,
        'booster':booster,
        'objective':objective_ml,
        'verbosity':verbosity,
        'tree_method':tree_method, 
        'predictor':predictor, 
        'eval_metric':eval_metric,
        'grow_policy': grow_policy,    
        # Specific sklearn api variables   
        'random_state':random_state,
        'n_jobs':parallel_jobs,
        'n_estimators':n_estimators,
        'use_label_encoder':False}
        #Instantiate the XGB Estimator 
        xgb_estimator=XGBClassifier(**params_cv)
        #Obtain the cross validation score - to be used by the trial to rate models
        cv_score=cross_val_score(xgb_estimator, X_train, y_train,scoring='f1',cv=skfolds,n_jobs=parallel_jobs).mean()    
        return cv_score
    #Create the Study
    study = optuna.create_study(study_name='XGB',direction='maximize',sampler=TPESampler(consider_magic_clip=True,seed=random_state,multivariate=True))
    # perform the search
    n_trials=360
    study.optimize(objective, n_trials=n_trials,n_jobs=parallel_jobs)
    4 replies
    A.G.
    @Divide-By-0
    Is there a way to have optuna print the hyperparam importances on plotly to a specific localhost port? right now when i do fig = optuna.visualization.plot_param_importances(study) fig.show(), it chooses localhost:<randomport>
    or at least print the port at which it prints to console, so I can ssh forward it
    A.G.
    @Divide-By-0
    I managed to have fig.show output to an svg, so resolved
    Is there a way to set optuna default hyperparams; i.e. optuna/optuna#1855, so at the start of an empty study it runs a trial with those hyperparams which I know are pretty good, and then searches knowing theres some pretty decent point there? would that even help the algorithm
    1 reply
    Hiroyuki Vincent Yamazaki
    @hvy

    As always, thanks for all the feedback and contributions. We’ve just released v2.9.0 with new features and code refactoring of the TPE sampler.

    🆕 Optuna from the command line without writing Python
    🆕 Weights & Biases integration
    🔨 TPE sampler refactorings

    Check out the highlights and release notes at
    https://github.com/optuna/optuna/releases/tag/v2.9.0 or with the Tweet https://twitter.com/OptunaAutoML/status/1422086080073854983 .

    Hiroyuki Vincent Yamazaki
    @hvy

    We just released a patch, 2.9.1.

    It includes changes to the Ask-and-Tell CLI subcommands to exclude storage URIs from the logs since those could contain sensitive information. Please consider upgrading if you are using these subcommands.
    https://github.com/optuna/optuna/releases/tag/v2.9.1

    Sree Aurovindh Viswanathan
    @sreeaurovindh
    I am running Optuna on Pytorch DDP with Mulitple GPUs. When i just use one GPU/ rank=0, the code runs fine. When I try with multi gpu, the code stops/freezes at trial.report(loss,epoch) . I am using https://github.com/optuna/optuna-examples/blob/main/pytorch/pytorch_distributed_simple.py#L41 as example. Any comments on how i can debug in my situation.
    1 reply
    TSHadley
    @tshadley
    Is there a way to use Optuna to minimize parameters? (Parameters, referring to coefficients of the network, not hyperparameters)
    1 reply
    Miguel Crispim Romao
    @romanovzky
    Hi all. Could you tell me what is the computational complexity of NSGAIISampler? I know that TPE is O(n^3) as any other GP, but I'm witnessing a massive slowdown as one approaches de 1000 trials with NSGAIISampler and I don't know why. Cheers!
    2 replies
    Zahra Taheri
    @zahta
    Hi everyone.
    My question is about using TPE sampler with Hyperband pruner from Optuna:
    Q1: What is the exact relation between study’s trials and brackets of Hyperband pruner? For example, if the size of a bracket is 10 (i.e. it contains 10 hyperparameter configurations), how many trials are executed for this bracket? Does Optuna consider a trial for each hyperparameter configuration?
    Q2: We set the number of epochs in training a neural network as the maximum resource of the Hyperband pruner. Based on the maximum resource value, are the number of epochs for training in each trial different?
    Q3: Does Optuna use maximum resource for the first and a few beginning Trials?
    Miguel Crispim Romao
    @romanovzky

    Hi all. Could you tell me what is the computational complexity of NSGAIISampler? I know that TPE is O(n^3) as any other GP, but I'm witnessing a massive slowdown as one approaches de 1000 trials with NSGAIISampler and I don't know why. Cheers!

    I wonder if there's any reason why the trials become progressively slower? This has been observed with RandomSampler too

    1 reply
    anshugarg1
    @anshugarg1
    Hello Everyone, I am trying to use Optuna in a project which uses both Pytorch Lightning (for code organisation) anf MLFlow (for logs tracking). So, out of 10 trials only last trail is getting stored my MLFLow. But ideally I expect it to store all the trials. Has someone ever done this before? It would be really helpful if someone can provide some inputs.
    3 replies
    juliangall
    @juliangall
    Screenshot 2021-08-30 at 06.59.07.pngI'm trying to understand the new format for the pareto plot with 2 objectives (v2.9.1). In the example, there are two very pale points at about (0.015, 0.195). (N.B. vertical axis is minimised, horizontal is maximised.) These are obviously on the pareto front and, for my example I would class them as the best points. How is Optuna determining which point is best on a pareto plot as, objectively, every point on the front is potentially the best? Also, what do the blue and red coloured vertical bars represent? Is there any documentation?
    2 replies
    gksriharsha
    @gksriharsha
    Hello Everyone! I am new to contributing to Optuna. The issue #1618 requires to add comments but list of files in the thread already have the appropriate Raises comments present in them. Could anyone help me please? Thank you
    4 replies
    Ali Kayhan Atay
    @AliKayhanAtay
    hey everyone, can someone explain me how can i supply LightGBMPruningCallback in to the LightGBMTunerCV ? i couldnt figure out since callback expects trials Thank you
    2 replies
    Basil Kraft
    @bask0

    Hi everyone! After having successfully tuned HPs using optuna, I want to run 10 different trials using the best HPs, each with a different configuration. I manipulate the best trial and add it to the study like this:

    best_trial = study.best_trial
    
    for fold in range(10):
        cv_trial = best_trial.copy()
        cv_trial.params['fold'] = fold
        study.add_trial(cv_trial)

    Now, if I call this in parallel, the same folds get evaluated multiple times. Is there an elegant way to deal with this, i.e., to only run each trial/fold once?

    3 replies
    Adrien RUAULT
    @ruaultadrien
    Hi everyone! I was wondering if it was possible to control the level of exploration of the TPESampler. Indeed I am running a 100 trials including 50 random trial. My issue is that after the 50th trial, the Sampler almost stops exploring. It always stays in the same range of hyperparameters. From my understanding of baysian optimization it should be possible to control the balance between exploitation and exploration. However I don't know if that is possible with TPE. Do you have any insihgts? :heart:
    2 replies
    JakobEliasWagner
    @JakobEliasWagner
    Hello Everyone :) I am looking for a way to test multiple kernel initializers and how well they work. These kernel initializers do not only have different names, but do have different parameters as well. What is the best way to optimiz those kinds of problems? Is it possible to create nested trial-parameters? Thank you
    3 replies
    Crissman Loomis
    @Crissman

    As always, thanks for all the feedback and contributions. We’ve just released v2.10.0 with new features and code refactoring.

    🆕 CLI tools for listing Optuna trials
    🆕 Multi-objective optimization support of Weights & Biases and MLflow integration

    Check out the highlights and release notes at
    https://github.com/optuna/optuna/releases/tag/v2.10.0 or with the Tweet https://twitter.com/OptunaAutoML/status/1444915450937176064.

    SungJun Cho
    @scho97
    Hello all! I was wondering how the parallelization is implemented in Bayesian optimization (BO) which uses samplers like TPE (and GP-EI). Optuna supports process-based optimization (link), and it seems like the trial at one node or GPU has to finish in order to run the trial at another node or GPU. I believe this is compatible to BO, since the result of previous trial is necessary to make predictions about the next sampling point, but I was not sure what advantages parallelization gives over using one computing source if trials are computed serially. Thank you!
    4 replies
    Anurag Gupta
    @anurag2896
    Hello everyone!
    I am new to optuna, and was looking to find some examples to implement unsupervised ML algorithms using optuna for their hyperparameter tuning, (Kmeans, etc), but I couldn't find any online. Can anyone help me with any link or their own example?
    6 replies
    Paras Koundal
    @ParasKoundal
    Hi, I am working with Graph Neural Networks. Was trying to use optuna to improve. However, I get the error Trial 0 failed, because the value None could not be cast to float. The only reply to this issue on github doesn't help
    1 reply
    robot.elbow
    @robot.elbow:matrix.org
    [m]

    Hi there =)

    I have been using Optuna for a bit (great tool big thanks!) - just started trying to combine it with Hydra using the Optuna pluging (https://hydra.cc/docs/next/plugins/optuna_sweeper/) - working well and succesfully tested storing results in an sqlite database. However I am struggling to get the Optuna-Dashboard to load results / to see the sqlite service - to note as well the python command study = optuna.load_study("test_study","sqlite:///test_study.db") raises a KeyError(NOT_FOUND_MSG) error - despite sqlite:///test_study.db storage target being sucessfully used by the Optuna Sweeper plugin...

    I am running this all in a docker container with a python:3.9.7-slim base...

    any thoughts / advice / direction for next step(s) would be greatly appreciated!!

    marz2
    @marz2:matrix.org
    [m]
    Hi. Is it possible in Optuna to report multiple metrics and do optimization only on one? I've been looking trough docs up and down but cannot find it.
    marz2
    @marz2:matrix.org
    [m]
    Or is the only option to use MLFlow?
    marz2
    @marz2:matrix.org
    [m]
    Oh, I see it's possible with trial.set_user_attr!