PROJECT TITLE :

Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints

ABSTRACT:

The momentum technique has only relatively recently emerged as a useful strategy for accelerating the convergence of gradient descent (GD) methods. It also demonstrates improved performance in both Deep Learning and regularized learning. Examples of typical momentum include Nesterov's accelerated gradient (NAG) and heavy-ball (HB) methods. [Citation needed] However, as of yet, the majority of the acceleration studies have only been conducted on NAG, and there have only been a few reports of investigations concerning the acceleration of HB. In this piece, we discuss individual convergence, which refers to the process of reaching a solution in nonsmooth optimizations with constraints by focusing on the final iteration of the HB algorithm. This question is important in the field of Machine Learning because it focuses on the constraints that must be imposed on the learning structure and the individual output that must be obtained in order to successfully guarantee this structure while also maintaining an optimal rate of convergence. To be more specific, we show that HB reaches an individual convergence rate of O(1/sqrt t), where t is the number of iterations in the algorithm. This suggests that both of the two momentum methods have the potential to speed up the individual convergence of basic GD to its optimal state. Even for the convergence of averaged iterates, our solution avoids the disadvantages of the prior work by limiting the optimization problem to be unconstrained and limiting the number of iterations that are performed to be predefined. This is true even for the case where our solution converges on an optimal solution. This article's novel approach to analysis of convergence provides a clear understanding of how the HB momentum can accelerate individual convergence and reveals more insights about the similarities and differences in obtaining averaging and individual convergence rates. Additionally, the article reveals more insights into how the HB momentum can accelerate individual convergence. The projection-based operation can be used to generate an individual solution in any setting, including those that are regularized and stochastic, so the optimal individual convergence that was derived for those settings can now be applied. When compared to the averaged output, the sparsity, on the other hand, can be significantly reduced without compromising the theoretical optimal rates. The effectiveness of the HB momentum strategy has been demonstrated by a number of real experiments.