Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ressources/demo/PEPit_demo_algorithm_design.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,16 @@
"For instance, consider the optimization problem\n",
"$$\\min_{x\\in\\mathbb{R}^d} f(x),$$\n",
"where $f$ is assumed to belong to some function class $\\mathcal{F}$. Assuming the gradients of $f$ are readily available, one may consider the following family of first-order algorithms parametrized by weights $\\{h_{i,j}\\}$:\n",
"$$\\begin{aligned}\n",
"\\begin{equation*}\n",
"\\begin{aligned}\n",
"y_1 &= y_0 - h_{1,0} \\nabla f(y_0),\\\\\n",
"y_2 &= y_1 - h_{2,0} \\nabla f(y_0) - h_{2,1} \\nabla f(y_1),\\\\\n",
"y_3 &= y_2 - h_{3,0} \\nabla f(y_0) - h_{3,1} \\nabla f(y_1) - h_{3,2} \\nabla f(y_2),\\\\\n",
"&\\vdots\\\\\n",
"y_N &= y_{N-1} - \\sum_{i=0}^{N-1} h_{N,i} \\nabla f(y_i),\n",
"\\end{aligned} \\tag{1}\\label{eq:ogm_start}$$\n",
"\\end{aligned}\n",
"\\tag{1}\\label{eq:ogm_start}\n",
"\\end{equation*}\n",
"and aim to choose $\\{h_{i,j}\\}$ as a solution to the minimax problem:\n",
"$$\\min_{\\{h_{i,j}\\}}\\,\\, \\max_{f\\in\\mathcal{F}_{0,L}} \\left\\{ \\frac{f(y_N)-f_\\star}{\\|y_0-x_\\star\\|^2_2} \\, : \\, y_N \\text{ obtained from } \\eqref{eq:ogm_start} \\text{ and } y_0 \\right\\}. \\tag{2}\\label{eq:pep_opt_h}$$\n",
"One natural approach is to choose the step sizes by solving $\\eqref{eq:pep_opt_h}$. Of course, methods of the form $\\eqref{eq:ogm_start}$ are impractical in general, since they require keeping track of all previous gradients, and we will try to *factor* them into more convenient forms. This notebook illustrates the following approaches:\n",
Expand Down