Pacific Journal of Mathematics


 Ophelia Norton
 1 years ago
 Views:
Transcription
1 Pacific Journal of Mathematics MINIMIZATION OF FUNCTIONS HAVING LIPSCHITZ CONTINUOUS FIRST PARTIAL DERIVATIVES LARRY ARMIJO Vol. 16, No. 1 November 1966
2 PACIFIC JOURNAL OF MATHEMATICS Vol. 16, No. 1, 1966 MINIMIZATION OF FUNCTIONS HAVING LIPSCHITZ CONTINUOUS FIRST PARTIAL DERIVATIVES LARRY ARMIJO A general convergence theorem for the gradient method is proved under hypotheses which are given below. It is then shown that the usual steepest descent and modified steepest descent algorithms converge under the some hypotheses. The modified steepest descent algorithm allows for the possibility of variable stepsize. For a comparison of our results with results previously obtained, the reader is referred to the discussion at the end of this paper. Principal conditions* Let / be a realvalued function defined and continuous everywhere on E n (real Euclidean wspace) and bounded below E n. For fixed x 0 e E n define S(x 0 ) = {x: f(x) ^ f(x 0 )}. The function^ / satisfies: condition I if there exists a unique point x* e E n such that f(x*) = inf/(a); Condition II at x 0 if fe C 1 on S(x 0 ) and Vf(x) = 0 xee n for x e S(x 0 ) if and only if x = x*; Condition III at x 0 if /e C 1 on S(x 0 ) and Ff is Lipschitz continuous on S(x 0 ), i.e., there exists a Lipschitz constant K> 0 such that Ff(y) Ff(x) \ S K\y x\ for every pair x, yes(x 0 ); Condition IV at x 0 if fec 1 on S(x 0 ) and if r > 0 implies that m(r) > 0 where ra(r) = inf Ff(x), S r (x Q ) = S r Π S(x 0 ), S r = χes r (χ 0 ) {x : I x x* I ^r}, and #* is any point for which f(x*) inf f(x). (If xee n S r (x 0 ) is void, we define m(r) = oo.) It follows immediately from the definitions of Conditions I through IV that Condition IV implies Conditions I and II, and if S(x 0 ) is bounded, then Condition IV is equivalent to Conditions I and II. 2. The convergence theorem* In the convergence theorem and its corollaries, we will assume that / is a realvalued function defined and continuous everywhere on E n, bounded below on E n, and that Conditions III and IV hold at x 0. THEOREM. // 0 < δ g 1/41SΓ, then for any xes(x 0 ), the set (1) S*(a, δ) = {x λ : x λ = x Wf{x), λ > 0, f(x λ )  f(x) ^  δ \Ff(x)\ 2 } Received January 30, The research for this paper was supported in part by General Dynamics/Astronautics, San Diego, California, Rice University, Houston, Texas, and the Martin Company, Denver, Colorado. The author is currently employed by the National Engineering Science Company, Houston, Texas.
3 2 LARRY ARMIJO is a nonempty subset of S(x 0 ) and any sequence {x k }ΐ= 0 such that x k+1, d), k = 0, 1, 2,, converges to the point x* which minimizes f. Proof. If xe S(x 0 ), x λ = x  Wf(x) and 0 ^ λ ^ 1/Z", Condition III and the mean value theorem imply the inequality f(x λ ) f(x) ^  (λ  X 2 K) I Pf(x) 2 which in turn implies that x λ e S*(x, δ) for λ x ^ λ ^ λ 2, λ< so that S*(x, d) is a nonempty subset of S(x Q ). If {^fc )Γ=o is any sequence for which x k+1, δ), k =0,1, 2,, then (1) implies that sequence {f(x k )}~= 0, which is bounded below, is monotone nonincreasing and hence that Vf(x k ) \ * 0 as k > oo. The remainder of the theorem follows from Condition IV. COROLLARY 1. (The Steepest Descent Algorithm) If x k+1 = x k  Lj7/(a? 4 ), k = 0, 1, 2, Δίί. then the sequence {x k }^=Q converges to the point x* which minimizes f. Proof. It follows from the proof of the convergence theorem that the sequence {x k }^= 0 defined in the statement of Corollary 1 is such that x k+1, 1/4Z), k = 0,1, 2,.. COROLLARY 2. (The Modified Steepest Descent Algorithm) If a is an arbitrarily assigned positive number, a m = a/2 m ~\ m = 1, 2,, and x k+1 = x k cc mj Pf(x k ) where m k is the smallest positive integer for which ( 2) f(x k  a m /f(x k ))  f(x k ) ^  \a mh \ Ff(x k ) 2, k = 0, 1, 2,, then the sequence {x k }ΐ= 0 converges to the point x* which minimizes f. Proof. It follows from the proof of the convergence theorem that if x e S(x Q ) and x λ =x  \Pf(x), then f(x λ )  f(x) ^  (1/2) λ Vf(x) 2 for 0 ^ λ ^ 1J2K. If a ^ l/2iγ, then for the sequence {x k }ΐ= 0 in the statement of Corollary 2, m k = 1 and x k+1, (l/2)a), k = 0,1, 2,. If a > 1/2Z", then the integers m^ exist and a mfc > l/4iγ so that
4 MINIMIZATION OF FUNCTIONS 3 3* Discussion* The convergence theorem proves convergence under hypotheses which are more restrictive than those imposed by Curry [1] but less restrictive than those imposed by Goldstein [2]. However, both the algorithms which we have considered would be considerably easier to apply than the algorithm proposed by Curry since his algorithm requires the minimization of a function of one variable at each step. The method of Goldstein requires the assumption that fec 2 on S(x 0 ) and that S(x 0 ) be bounded. It also requires knowledge of a bound for the norm of the Hessian matrix of / on S(x 0 ), but yields an estimate for the ultimate rate of convergence of the gradient method. It should be pointed out that the modified steepest descent algorithm of Corollary 2 allows for the possibility of variable stepsize and does not require knowledge of the value of the Lipschitz constant K. The author is indebted to the referee for his comments and suggestions. REFERENCES 1. H. B. Curry, The method of steepest descent for nonlinear minimization problems, Quart. Appl. Math. 2 (1944), '2. A. A. Goldstein, Cauchy's method of minimization, Numer. Math. 4 (2), (1962),
5
6 PACIFIC JOURNAL OF MATHEMATICS H. SAMELSON Stanford University Stanford, California R. M. BLUMENTHAL University of Washington Seattle, Washington EDITORS *J. DUGUNDJI University of Southern California Los Angeles, California RICHARD ARENS University of California Los Angeles, California ASSOCIATE EDITORS E. F. BECKENBACH B. H. NEUMANN F. WOLF K. YOSIDA UNIVERSITY OF BRITISH COLUMBIA CALIFORNIA INSTITUTE OF TECHNOLOGY UNIVERSITY OF CALIFORNIA MONTANA STATE UNIVERSITY UNIVERSITY OF NEVADA NEW MEXICO STATE UNIVERSITY OREGON STATE UNIVERSITY UNIVERSITY OF OREGON OSAKA UNIVERSITY UNIVERSITY OF SOUTHERN CALIFORNIA SUPPORTING INSTITUTIONS STANFORD UNIVERSITY UNIVERSITY OF TOKYO UNIVERSITY OF UTAH WASHINGTON STATE UNIVERSITY UNIVERSITY OF WASHINGTON * * * AMERICAN MATHEMATICAL SOCIETY CHEVRON RESEARCH CORPORATION TRW SYSTEMS NAVAL ORDNANCE TEST STATION Printed in Japan by International Academic Printing Co., Ltd., Tokyo Japan
7 Pacific Journal of Mathematics Vol. 16, No. 1 November, 1966 Larry Armijo, Minimization of functions having Lipschitz continuous first partial derivatives Edward Martin Bolger and William Leonard Harkness, Some characterizations of exponentialtype distributions James Russell Brown, Approximation theorems for Markov operators Doyle Otis Cutler, Quasiisomorphism for infinite Abelian pgroups Charles M. Glennie, Some identities valid in special Jordan algebras but not valid in all Jordan algebras Thomas William Hungerford, A description of Mult i (A 1,, A n ) by generators and relations James Henry Jordan, The distribution of cubic and quintic nonresidues Junius Colby Kegley, Convexity with respect to EulerLagrange differential operators Tilla Weinstein, On the determination of conformal imbedding Paul Jacob Koosis, On the spectral analysis of bounded functions JeanPierre Kahane, On the construction of certain bounded continuous functions V. V. Menon, A theorem on partitions of massdistribution Ronald C. Mullin, The enumeration of Hamiltonian polygons in triangular maps Eugene Elliot Robkin and F. A. Valentine, Families of parallels associated with sets Melvin Rosenfeld, Commutative Falgebras A. Seidenberg, Derivations and integral closure S. Verblunsky, On the stability of the set of exponents of a Cauchy exponential series Herbert Walum, Some averages of character sums
Pacific Journal of Mathematics
Pacific Journal of Mathematics INJECTIVE MODULES OVER NOETHERIAN RINGS EBEN MATLIS Vol 8, No 3 May 1958 INJECTIVE MODULES OVER NOETHERIAN RINGS EBEN MATLIS Introduction In this discussion every module
More informationProperties of BMO functions whose reciprocals are also BMO
Properties of BMO functions whose reciprocals are also BMO R. L. Johnson and C. J. Neugebauer The main result says that a nonnegative BMOfunction w, whose reciprocal is also in BMO, belongs to p> A p,and
More informationHow to Use Expert Advice
NICOLÒ CESABIANCHI Università di Milano, Milan, Italy YOAV FREUND AT&T Labs, Florham Park, New Jersey DAVID HAUSSLER AND DAVID P. HELMBOLD University of California, Santa Cruz, Santa Cruz, California
More informationTHE PROBLEM OF finding localized energy solutions
600 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 3, MARCH 1997 Sparse Signal Reconstruction from Limited Data Using FOCUSS: A Reweighted Minimum Norm Algorithm Irina F. Gorodnitsky, Member, IEEE,
More informationControllability and Observability of Partial Differential Equations: Some results and open problems
Controllability and Observability of Partial Differential Equations: Some results and open problems Enrique ZUAZUA Departamento de Matemáticas Universidad Autónoma 2849 Madrid. Spain. enrique.zuazua@uam.es
More informationDecoding by Linear Programming
Decoding by Linear Programming Emmanuel Candes and Terence Tao Applied and Computational Mathematics, Caltech, Pasadena, CA 91125 Department of Mathematics, University of California, Los Angeles, CA 90095
More informationOptimization by Direct Search: New Perspectives on Some Classical and Modern Methods
SIAM REVIEW Vol. 45,No. 3,pp. 385 482 c 2003 Society for Industrial and Applied Mathematics Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods Tamara G. Kolda Robert Michael
More informationA Modern Course on Curves and Surfaces. Richard S. Palais
A Modern Course on Curves and Surfaces Richard S. Palais Contents Lecture 1. Introduction 1 Lecture 2. What is Geometry 4 Lecture 3. Geometry of InnerProduct Spaces 7 Lecture 4. Linear Maps and the Euclidean
More informationSubspace Pursuit for Compressive Sensing: Closing the Gap Between Performance and Complexity
Subspace Pursuit for Compressive Sensing: Closing the Gap Between Performance and Complexity Wei Dai and Olgica Milenkovic Department of Electrical and Computer Engineering University of Illinois at UrbanaChampaign
More informationA GRAPHTHEORETIC DEFINITION OF A SOCIOMETRIC CLIQUE *
Journal of Mathematical Sociology Gordon and Breach Science Publishers 1973 Vol. 3, pp 113126 Printed in Birkenhead, England A GRAPHTHEORETIC DEFINITION OF A SOCIOMETRIC CLIQUE * RICHARD D. ALBA Columbia
More informationIEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 4, APRIL 2006 1289. Compressed Sensing. David L. Donoho, Member, IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 4, APRIL 2006 1289 Compressed Sensing David L. Donoho, Member, IEEE Abstract Suppose is an unknown vector in (a digital image or signal); we plan to
More informationHow many numbers there are?
How many numbers there are? RADEK HONZIK Radek Honzik: Charles University, Department of Logic, Celetná 20, Praha 1, 116 42, Czech Republic radek.honzik@ff.cuni.cz Contents 1 What are numbers 2 1.1 Natural
More informationRECENTLY, there has been a great deal of interest in
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 47, NO. 1, JANUARY 1999 187 An Affine Scaling Methodology for Best Basis Selection Bhaskar D. Rao, Senior Member, IEEE, Kenneth KreutzDelgado, Senior Member,
More informationThe Backpropagation Algorithm
7 The Backpropagation Algorithm 7. Learning as gradient descent We saw in the last chapter that multilayered networks are capable of computing a wider range of Boolean functions than networks with a single
More information= 2 + 1 2 2 = 3 4, Now assume that P (k) is true for some fixed k 2. This means that
Instructions. Answer each of the questions on your own paper, and be sure to show your work so that partial credit can be adequately assessed. Credit will not be given for answers (even correct ones) without
More informationRichard E. Zinbarg northwestern university, the family institute at northwestern university. William Revelle northwestern university
psychometrika vol. 70, no., 23 33 march 2005 DOI: 0.007/s33600309747 CRONBACH S α, REVELLE S β, AND MCDONALD S ω H : THEIR RELATIONS WITH EACH OTHER AND TWO ALTERNATIVE CONCEPTUALIZATIONS OF RELIABILITY
More informationSOME METHODS FOR CLASSIFICATION AND ANALYSIS
SOME METHODS FOR CLASSIFICATION AND ANALYSIS OF MULTIVARIATE OBSERVATIONS J. MACQUEEN UNIVERSITY OF CALIFORNIA, Los ANGELES 1. Introduction The main purpose of this paper is to describe a process for partitioning
More informationThe Capital Asset Pricing Model: Some Empirical Tests
The Capital Asset Pricing Model: Some Empirical Tests Fischer Black* Deceased Michael C. Jensen Harvard Business School MJensen@hbs.edu and Myron Scholes Stanford University  Graduate School of Business
More informationTwoSided Matching Theory
TwoSided Matching Theory M. Utku Ünver Boston College Introduction to the Theory of TwoSided Matching To see which results are robust, we will look at some increasingly general models. Even before we
More informationCommunication Theory of Secrecy Systems
Communication Theory of Secrecy Systems By C. E. SHANNON 1 INTRODUCTION AND SUMMARY The problems of cryptography and secrecy systems furnish an interesting application of communication theory 1. In this
More informationLaw Invariant Risk Measures have the Fatou Property
Law Invariant Risk Measures have the Fatou Property E. Jouini W. Schachermayer N. Touzi Abstract S. Kusuoka [K 1, Theorem 4] gave an interesting dual characterization of law invariant coherent risk measures,
More informationHighRate Codes That Are Linear in Space and Time
1804 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 48, NO 7, JULY 2002 HighRate Codes That Are Linear in Space and Time Babak Hassibi and Bertrand M Hochwald Abstract Multipleantenna systems that operate
More informationSome Applications of Laplace Eigenvalues of Graphs
Some Applications of Laplace Eigenvalues of Graphs Bojan MOHAR Department of Mathematics University of Ljubljana Jadranska 19 1111 Ljubljana, Slovenia Notes taken by Martin Juvan Abstract In the last decade
More informationSome Research Problems in Uncertainty Theory
Journal of Uncertain Systems Vol.3, No.1, pp.310, 2009 Online at: www.jus.org.uk Some Research Problems in Uncertainty Theory aoding Liu Uncertainty Theory Laboratory, Department of Mathematical Sciences
More informationNearOptimal Sensor Placements in Gaussian Processes
Carlos Guestrin Andreas Krause Ajit Paul Singh School of Computer Science, Carnegie Mellon University GUESTRIN@CS.CMU.EDU KRAUSEA@CS.CMU.EDU AJIT@CS.CMU.EDU Abstract When monitoring spatial phenomena,
More informationMatthias Beck Gerald Marchesi Dennis Pixton Lucas Sabalka
Matthias Beck Gerald Marchesi Dennis Pixton Lucas Sabalka Version.5 Matthias Beck A First Course in Complex Analysis Version.5 Gerald Marchesi Department of Mathematics Department of Mathematical Sciences
More informationA new continuous dependence result for impulsive retarded functional differential equations
CADERNOS DE MATEMÁTICA 11, 37 47 May (2010) ARTIGO NÚMERO SMA#324 A new continuous dependence result for impulsive retarded functional differential equations M. Federson * Instituto de Ciências Matemáticas
More informationf (x) π (dx), n f(x k ). (1) k=1
To appear Annals of Applied Probability 2005Vol. 0, No. 0, 1 8 ON THE ERGODICITY PROPERTIES OF SOME ADAPTIVE MCMC ALGORITHMS By Christophe Andrieu, and Éric Moulines University of Bristol and École Nationale
More informationGiotto: A TimeTriggered Language for Embedded Programming
Giotto: A TimeTriggered Language for Embedded Programming THOMAS A HENZINGER, MEMBER, IEEE, BENJAMIN HOROWITZ, MEMBER, IEEE, AND CHRISTOPH M KIRSCH Invited Paper Giotto provides an abstract programmer
More informationWhich Problems Have Strongly Exponential Complexity? 1
Journal of Computer and System Sciences 63, 512 530 (2001) doi:10.1006/jcss.2001.1774, available online at http://www.idealibrary.com on Which Problems Have Strongly Exponential Complexity? 1 Russell Impagliazzo
More information