Different input for optimization strategy

 

• Zero-order oracle: returns the function value f (x).

• First-order oracle: returns the function value f (x) and the gradient ∇f (x).

• Second-order oracle: returns f (x), ∇f (x), and the Hessian ∇ 2 f (x).

1.2 Local methods, unconstrained

1.3 first order methods

gradient and newton methods are first and second order methods:

different convergence rate: newton(quadratic) vs gradient(linear)

conjugate gradient:= (BFGS most stable)

constrained optimization,

• Lagrange relaxation, supinf <= infsup,
• penalty function

• barrier function, which goes to infinity at the bound

Dual

convex optimization only when f is concave