Notes on Convex Optimization

By -
1.1 Sub-level sets.
- Definition: The $\alpha$-sublevel set of a function $f$ contains all the feasible input $x$ of $f$ having the value $f(x)<=\alpha$.

- Properties: Sublevel sets of a convex function are convex for all $\alpha$ ($f(x) \leq \alpha$, $f(y) \leq \alpha$, $f(\theta x + (1-\theta) y) \leq \alpha $). If $f$ is concave, then its $\alpha$-superlevel set is a convex.

- Usage: To establish convexity of a convex set by associating it as a sublevel set of a convex function or a superlevel set of a concave function.

1.2 Epi-graph.
- Definition: epi f = $\{ (x, t) | x \in dom f, f(x) \leq t \}$. Epi means 'above'.

- Properties: A function is convex iff its epigraph is a convex set. A function is concave iff its hypograph is a convex set.

- Usage: Many results for convex functions can be proved (or interpreted) geometrically using epigraphs, and applying results for convex sets.

1.3 Closed set. 
- Definition: A set is closed if it contains its boundary.

1.4 Closed function: 
- Definition: a function is closed if its epi-graph is a closed set.

1.5 Conjugate functions.
- Definition: Let f: $R^n \rightarrow R$, $f^*:R^n \rightarrow R$ is defined
$$ f^*(y) = sup_{x\in dom f}(y^Tx - f(x))$$
- Properties:
   + $f^*$ is a convex function whether f is or is not convex.
   + Fenchel's inquality: $f(x) + f^*(y) \geq x^Ty$
   + Second conjugate: if $f$ is convex, $f$ is closed, then $f^{**} = f$.
   + Legendre transform: the conjugate of a differentiable function $f$ is also called the Legendre transform of $f$. Let $z \in R^n$ be arbitrary and define $y = \nabla f(z)$, we have $f^*(y) = z^T \nabla f(z) - f(z)$.
   + Scaling and composition with affine transformation.
   + Sums of independent functions: if $f(u, v) = f_1(u) + f_2(v)$, $f_1$, $f_2$ are convex functions with conjugates $f^*_1$ and $f^*_2$, then $f^*(u, v) = f^*_1(u) + f^*_2(v)$.

1.6 Strong convexity and implications
- Definition: an objective function is strongly convex on $S$, which means there exists an $m > 0$ such that $\nabla^2 f(x) \succeq m I$, for all $x \in S$.
- Properties:
   + For $x, y \in S$, we have $f(y) = f(x) + \nabla f(x)^T (y-x) + \frac{1}{2}(y-x)^T\nabla^2 f(z) (y-x)$, for some $z$ on the line segment $[x, y]$.
   + By the strong convexity assumption, the last term on the RHS is at least $(m/2)\|y-x \|^2_2$. So we have the inequality $f(y) \geq f(x) + \nabla f(x)^T (y-x) + \frac{m}{2}\|y-x\|^2_2$, for all x and y in $S$. When $m = 0$, we recover the basic inequality characterizing convexity; for $m > 0$, we obtain a better lower bound on $f(y)$ than follows from convexity alone.


Comments

Post a Comment

Popular posts from this blog

Manifold Learning [unfinished]

By -
Image
I. What is manifold learning Dimensionality Reduction The accuracy of the training algorithms is directly proportional to the amount of data we have. However, managing a large number of features is usually a burden to our algorithm.  Some of these features may be irrelevant, so it's important to make sure that the final model doesn't get affected by this.  What is Dimensionality?  Dimensionality refers to the minimum number of coordinates needed to specify any point within a space or an object. Why do we need Dimensionality Reduction? If you keep the dimensionality high, it will be nice and unique but it may not be easy to analyse because of complexity involved. Apart from simplifying data, visualization is also an interesting and challenging application. Linear Dimensionality Reduction Fig 1 : PCA illustration (source: http://evelinag.com/blog ) Principle Component Analysis  Given a data set, PCA finds the directions along which the d...
Read more

Find all pairs in array of integers whose sum is equal to a given number ?

By -
Problem: Write a program to find all pairs of integers whose sum is equal to a given number. For example if input integer array is {1, 2, 3, 4, 5}, and the given sum is 5, the code should return two pairs {1, 4} and {2, 3}. This is a common interview question related to array structure. A quick search on Google can return several pages discussing the problem, to list but a few [1,2, 3, 4]. Those pages only discuss the solutions at the conceptual level, without supporting comparing numbers, although they also provide the code. In this note I would re-implement all these approaches and compare their efficiency in practice. Common approaches There are four approaches which are often mentioned together: Brute-force, HashSet, sorting with search and sorting with binary search. Each has its own pros and cons. Interested readers can find the details in the list of sources below. Here is a summary of the storage complexity and computational complexity. Approach Storage co...
Read more

Feature scaling

By -
Image
1. What is feature scaling?  Feature scaling is a method used to normalize all independent values of our data. We also call it as data normalization and is generally performed before running machine learning algorithms. 2. Why do we need to use feature scaling?  In practice the range of raw data is very wide and hence the object functions will not work properly (means that it will stuck at local optimum), or they are time-consuming without normalization. For example:  K-Means, might give you totally different solutions depending on the preprocessing methods that you used. This is because an affine transformation implies a change in the metric space: the Euclidean distance between two samples will be different after that transformation. When we apply gradient descent, feature scaling also helps it to converge much faster that without normalized data.  With and without feature scaling in gradient descent 3. Methods in feature scaling? Rescaling The simp...
Read more