Downloads & Free Reading Options - Results

Set-functions appear in many areas of computer science and applied mathematics, such as machine learning, computer vision, operations research or electrical networks. Among these set-functions, submodular functions play an important role, similar to convex functions on vector spaces. In this tutorial, the theory of submodular functions is presented, in a self-contained way, with all results shown from first principles. A good knowledge of convex analysis is assumed.

This paper considers the problem of minimizing the time average of a stochastic process subject to time average constraints on other processes. A canonical example is minimizing average power in a data network subject to multi-user throughput constraints. Another example is a (static) convex program. Under a Slater condition, the drift-plus-penalty algorithm is known to provide an $O(\epsilon)$ approximation to optimality with a convergence time of $O(1/\epsilon^2)$. This paper proves the same result with a simpler technique and in a more general context that does not require the Slater condition. This paper also emphasizes application to basic convex programs, linear programs, and distributed optimization problems.

This paper considers the problem of minimizing the time average of a stochastic process subject to time average constraints on other processes. A canonical example is minimizing average power in a data network subject to multi-user throughput constraints. Another example is a (static) convex program. Under a Slater condition, the drift-plus-penalty algorithm is known to provide an $O(\epsilon)$ approximation to optimality with a convergence time of $O(1/\epsilon^2)$. This paper proves the same result with a simpler technique and in a more general context that does not require the Slater condition. This paper also emphasizes application to basic convex programs, linear programs, and distributed optimization problems.

We extend a multi-agent convex-optimization algorithm named Newton-Raphson consensus to a network scenario that involves directed, asynchronous and lossy communications. We theoretically analyze the stability and performance of the algorithm and, in particular, provide sufficient conditions that guarantee local exponential convergence of the node-states to the global centralized minimizer even in presence of packet losses. Finally, we complement the theoretical analysis with numerical simulations that compare the performance of the Newton-Raphson consensus against asynchronous implementations of distributed subgradient methods on real datasets extracted from open-source databases.

We combine resolvent-mode decomposition with techniques from convex optimization to optimally approximate velocity spectra in a turbulent channel. The velocity is expressed as a weighted sum of resolvent modes that are dynamically significant, non-empirical, and scalable with Reynolds number. To optimally represent DNS data at friction Reynolds number $2003$, we determine the weights of resolvent modes as the solution of a convex optimization problem. Using only $12$ modes per wall-parallel wavenumber pair and temporal frequency, we obtain close agreement with DNS-spectra, reducing the wall-normal and temporal resolutions used in the simulation by three orders of magnitude.

We address the problem of compressed sensing with multiple measurement vectors associated with prior information in order to better reconstruct an original sparse matrix signal. $\ell_{2,1}-\ell_{2,1}$ minimization is used to emphasize co-sparsity property and similarity between matrix signal and prior information. We then derive the necessary and sufficient condition of successfully reconstructing the original signal and establish the lower and upper bounds of required measurements such that the condition holds from the perspective of conic geometry. Our bounds further indicates what prior information is helpful to improve the the performance of CS. Experimental results validates the effectiveness of all our findings.

In this paper, the core convex topology on a real vector space $X$, which is constructed just by $X$ operators, is investigated. This topology, denoted by $\tau_c$, is the strongest topology which makes $X$ into a locally convex space. It is shown that some algebraic notions $(closure ~ and ~ interior)$ existing in the literature come from this topology. In fact, it is proved that algebraic interior and vectorial closure notions, considered in the literature as replacements of topological interior and topological closure, respectively, in vector spaces not necessarily equipped with a topology, are actually nothing else than the interior and closure with the respect to the core convex topology. We reconstruct the core convex topology using an appropriate topological basis which enables us to characterize its open sets. Furthermore, it is proved that $(X,\tau_c)$ is not metrizable when X is infinite-dimensional, and also it enjoys the Hine-Borel property. Using these properties, $\tau_c$-compact sets are characterized and a characterization of finite-dimensionality is provided. Finally, it is shown that the properties of the core convex topology lead to directly extending various important results in convex analysis and vector optimization from topological vector spaces to real vector spaces.

We propose a new class of convex penalty functions, called \emph{variational Gram functions} (VGFs), that can promote pairwise relations, such as orthogonality, among a set of vectors in a vector space. These functions can serve as regularizers in convex optimization problems arising from hierarchical classification, multitask learning, and estimating vectors with disjoint supports, among other applications. We study convexity for VGFs, and give efficient characterizations for their convex conjugates, subdifferentials, and proximal operators. We discuss efficient optimization algorithms for regularized loss minimization problems where the loss admits a common, yet simple, variational representation and the regularizer is a VGF. These algorithms enjoy a simple kernel trick, an efficient line search, as well as computational advantages over first order methods based on the subdifferential or proximal maps. We also establish a general representer theorem for such learning problems. Lastly, numerical experiments on a hierarchical classification problem are presented to demonstrate the effectiveness of VGFs and the associated optimization algorithms.

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method. We hope that the basic convergence results developed in this paper can serve the reference to the convergence of stochastic momentum methods and also serve the baselines for comparison in future development of stochastic momentum methods. The novelty of convergence analysis presented in this paper is a unified framework, revealing more insights about the similarities and differences between different stochastic momentum methods and stochastic gradient method. The unified framework exhibits a continuous change from the gradient method to Nesterov's accelerated gradient method and finally the heavy-ball method incurred by a free parameter, which can help explain a similar change observed in the testing error convergence behavior for deep learning. Furthermore, our empirical results for optimizing deep neural networks demonstrate that the stochastic variant of Nesterov's accelerated gradient method achieves a good tradeoff (between speed of convergence in training error and robustness of convergence in testing error) among the three stochastic methods.