I recently made a post to the Discourse forum about trying to find a way to access the approximated Hessian from a converged solution to use as the preconditioner on a future minimization of a slightly altered initial state; I'm using BFGS to bifurcate from previously minimizers. After looking at Optim.jl, libLBFGS, and LBFGS++ I couldn't find a interface, so I imagine that it is most likely not available in Optim.jl either. I wanted to double check here if that's something viable with the current interface, if there would be any interest in adding it to the package, or if I should try to extend this on my own in Julia somehow.
So far, the only package outside of Julia that implements this is SciPy:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_bfgs.html
state object
state should just be added to MultivariateOptimizationResults?
optimize(d::D, initial_x::Tx, method::M,
options::Options = Options(;default_options(method)...),
state = initial_state(method, options, d, initial_x))state it'll get mutated and you can get what you want from it after
Hi All, new here ... We have been using Optim.jl and LBFGS for a while, and for our particular problems the combination of alphaguess = LineSearches.InitialQuadratic() and linesearch = LineSearches.MoreThuente() works best. I thought I would try that for Rosenbrock on the front page, and it beats BFGS pretty handily, at least in terms of operation count, almost 2x. Thought that was worth mentioning.
Where is a pointer to the slack channel please? Cheers
fun(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
For example, we want to get the following Hessian from the above objective function.
function fun_hess!(h, x)
h[1, 1] = 2.0 - 400.0 x[2] + 1200.0 x[1]^2
h[1, 2] = -400.0 x[1]
h[2, 1] = -400.0 x[1]
h[2, 2] = 200.0
end
@ChrisRackauckas
@segawachobbies_twitter you can use ModelingToolkit to test it
using ModelingToolkit
@variables x[1:2]
result = ModelingToolkit.hessian((1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2, x)I tried the above code in ModelingToolkit and successfully got the ideal result! Thank you very much.
I compared Automatic differentiation with this method, and the difference in processing speed is about 100 times faster during optimization.
I think this is one of the options for fast optimization by just using the objective function.
using ModelingToolkit
using Optim
@variables x[1:2]
f = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
g = ModelingToolkit.gradient(f, x)
h = ModelingToolkit.hessian(f, x)
buildedF = build_function(f, x[1:2])
buildedG = build_function(g, x[1:2])
buildedH = build_function(h, x[1:2])
newF = eval(buildedF)
newG! = eval(buildedG[2])
newH! = eval(buildedH[2])
initial_x = zeros(2)
@time Optim.minimizer(optimize(newF, initial_x, Newton(); autodiff=:forward))
@time Optim.minimizer(optimize(newF, newG!, newH!, initial_x, Newton()))
I recently implemented the simulatenous stochastic perturbation algorthm (SPSA) and the model gradient descent (MDG, first developed in https://arxiv.org/abs/2005.11011) in a roughly Optim.jl compatible way. Is there interest in adding these methods to Optim.jl? If yes, I would make my code fully compatible to the Optim.jl interface and open a PR
