Stanford EE364A Convex Optimization I Stephen Boyd I 2023 I Lecture 18

Stanford EE364A Convex Optimization I Stephen Boyd I 2023 I Lecture 18

Convex Optimization

  • Convex problems can be solved efficiently and reliably, and have several advantages such as finding exact solutions and potential for embedding in control systems.
  • Many applications can be formulated as convex problems, but most real-world problems are not truly convex.
  • Convex optimization problems have identical structures across different fields and can be solved efficiently using tools like CVX PI.
  • Understanding the dual of a convex problem can provide insights and practical benefits.
  • Advanced techniques like repeatedly forming and solving convex approximations can be used to solve complex problems.

Sparse Signal Reconstruction

  • Sparse signal reconstruction involves finding a sparse representation of a signal from noisy measurements.
  • L1 reconstruction, also known as LASSO, is an effective method for sparse signal reconstruction.
  • Total variation denoising is an application of sparse signal reconstruction to image denoising.
  • Total variation regularization can be used to reduce noise while preserving sharp edges in images.
  • Iterated weighted L1 regularization can improve the performance of total variation regularization.

Cardinality Constraint

  • The cardinality constraint can be applied to various scenarios, such as piecewise constant fitting or piecewise linear fitting.
  • The L1 norm heuristic is commonly used as a convex relaxation for cardinality constrained problems.
  • Polishing is a technique where L1 heuristic is used to guess the sparsity pattern of the solution, and then the problem is resolved with only the non-zero entries.
  • L1 penalty keeps up the pressure to make the coefficients small all the way up until zero, which explains its effectiveness as a sparsifying regularizer.
  • There are other sparsifying penalties, such as the Huber penalty and the buru penalty, which combine different types of penalties to achieve sparsity.

Other Topics

  • Sparse solutions of linear inequalities can be found using the L1 heuristic, but it doesn't always find the global solution.
  • Time-varying AR models can be fit using a sparsity-inducing regularizer on the differences in the coefficients.
  • The dual of the spectral norm is the sum of the singular values and it can be used as a low-rank regularizer.
  • Sparse Bayesian models can be estimated by maximizing the log-likelihood minus the dual norm of the precision matrix.
  • Subgradient methods and stochastic subgradient methods are covered in other courses such as 364b.
  • Distributed convex optimization allows multiple parties to collaborate to solve a problem without sharing their data.

Overwhelmed by Endless Content?