Solving Very Large Financial Planning Problems on Blue Gene

Transcription

1 U N I V E R S School of Mathematics T H E O I T Y H F G E D I N U R Solving Very Large Financial Planning Problems on lue Gene ndreas Grothey, University of Edinburgh joint work with Jacek Gondzio, Marco Colombo urora 2007, Vienna 1

2 Overview sset and Liability Management Mean-Variance Formulation/Stochastic Programming Structure of Problem OOPS (Interior Point Solver) Interior Point Methods Structure Exploitation through Object Oriented Design Challenges in Massively Parallel Implementation Communications Data Management Granularity of Computations Further Developments urora 2007, Vienna 2

3 Portfolio Optimization: sset and Liability Management set of assets J = {1,..., J} is given (e.g. bonds, stock, real estate). t every stage t = 0,..., T 1 we can buy or sell different assets. The return of asset j at stage t is uncertain (but distribution is known). We have to make investment decisions: what to buy or sell, at which time stage Objectives: maximize the final wealth minimize the associated risk Extensions (leading to nonlinear models) } Mean Variance formulation: max IE(X) ρvar(x) different risk measures (one-sided, expected shortfall/value at risk) (nonlinear) utility functions urora 2007, Vienna 3

4 Modelling: Multiperiod Mean-Variance Model Variables: x h j,t position in asset j at time t. x s j,t,xb j,t amount of asset j bought/sold at time t. Parameters: v j value of asset j r j,t return of asset j when held at time t L t, C t liabilities/cash contributions at time t Objective: max IE(X) ρvar(x), X = (1 c t ) j v j x h j,t Constraints: C t + (1 c t ) j x h j,t = (1 + r t 1,j )x h j,t 1 x s j,t 1 + x b j,t 1 v j x s j,t = L t + (1+c t ) j v j x b j,t (inventory) (cash balance) urora 2007, Vienna 4

5 Modelling: Multiperiod Mean-Variance Model Variables: x h j,t position in asset j at time t. x s j,t,xb j,t amount of asset j bought/sold at time t. Parameters: v j value of asset j r j,t return of asset j when held at time t L t, C t liabilities/cash contributions at time t Objective: max IE(X) ρvar(x), X = (1 c t ) j v j x h j,t Constraints: C t + (1 c t ) j x h j,t = (1 + r t 1,j )x h j,t 1 x s j,t 1 + x b j,t 1 v j x s j,t = L t + (1+c t ) j v j x b j,t (inventory) (cash balance) urora 2007, Vienna 4

6 Stochastic Programming: use event tree (t-1,a(n)) ξ 1 ξ 2 (t,n-1) (t,n) Stages represent points in time at which decisions are taken. Nodes represent possible futures (given by asset returns and probabilities): r (t,n) j return of asset j at node (t, n). p (t,n) probability of reaching node (t, n). L (t,n), C (t,n) liability payment/cash-contribution at node (t, n). urora 2007, Vienna 5

7 sset and Liability Management as Stochastic Program With every node i = (t, n) we associate variables: x h j,i the position in asset j at node i; x b j,i, x s j,i the amount of asset j bought/sold at node i. and constraints: (1 + r i,j )x h π(i),j = xh i,j x b i,j + x s i,j, j (inventory) (1 + c t )v j x b i,j + L i = (1 c t )v j x s i,j + C i (cash balance) j j Objective is sum over final stage scenarios min IE(X) ρvar(x) = IE(X) ρie[(x E(X)) 2 ] = (1 c t ) p i v j x h i,j ρ p i [ ] i L T j i LT urora 2007, Vienna 6

8 Multistage Stochastic Programming Period 1 Period 2 Period 3 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario Tree Constraint Matrix Symmetrical event tree with K realizations at each node and T periods corresponds to K T 1 K T 1 scenarios nodes (blocks) K 1 urora 2007, Vienna 7

9 Structure of Objective IE(X) ρvar(x) = IE(X) ρ[ie(x 2 ) IE(X) 2 = [ p i v j x h ij ρ p i (v j x h ij )2 [ ] p i v j x h ij ]2 i L T j i L T j i L T j Dense, convex Hessian urora 2007, Vienna 8

10 Structure of Objective IE(X) ρvar(x) = IE(X) ρ[ie(x 2 ) IE(X) 2 = [ p i v j x h ij ρ p i (v j x h ij )2 [ ] p i v j x h ij ]2 i L T j i L T j i L T j lternatively: with y = p i v j x h ij i L T j IE(X) ρvar(x) = y ρ p i (v j x h ij) 2 y 2 i LT j Dense, convex Hessian Sparse, nonconvex Hessian 1 urora 2007, Vienna 8

11 LM: Structure of matrices and : Matrix Matrix 1 d d d d 1 urora 2007, Vienna 9

12 Stochastic Programming approach to Financial Planning Popular with academics, practitioners seem suspicious Large problem sizes Places constraints on model (?) Traditionally solved by decomposition Nested enders Decomposition aka L-shaped method Works well for LP models Unclear how to extend to quadratic/nonlinear models Or solved by Interior Point Methods Use taylored linear algebra Only certain types of constraints are allowable (?) Structure Exploitation is key to succesful implementation urora 2007, Vienna 10

13 OOPS - Object Oriented Parallel (Interior Point) Solver Mantra: Truly large scale problems are not only sparse but structured (due to e.g. dynamics, uncertainty, spatial distribution etc.) Exploiting structure is key to building efficient IPMs for large problems: Faster linear algebra Reduced memory use (by use of implicit factorization) Possibility to exploit (massive) parallelism We assume that structure is known! no automatic detection. OOPS is a general purpose (parallel) Interior Point solver Not tuned to any particular hardware Not tuned to any particular problem (structure). OOPS currently solves LP/P problems. NLP extension solves nonlinear financial planning problems urora 2007, Vienna 11

14 Interior Point Methods min c x x x s.t. x = b x 0 Optimality conditions: c + x y z = 0 x = b XZe = 0(µe) x, z 0 (P) Newton Step: I 0 0 Z 0 X x y z = ξ c ξ b r xz = c + x y z b x µe XZe (NS-P) Reduced to [ Θ 0 ] [ x y ] = [ ξc X 1 r xz ξ b ] where Θ = X 1 Z, X = diag(x), Z = diag(z) urora 2007, Vienna 12

15 Linear lgebra of IPMs Main work: solve [ Θ ] } {{ 0 } Φ [ x y ] = [ r h ] for several right-hand-sides at each iteration Two stage solution procedure factorize Φ = LDL backsolve(s) to compute direction ( x, y) + corrections Φ changes numerically but not structurally at each iteration Key to efficient implementation is exploiting structure of Φ in these two steps urora 2007, Vienna 13

16 Example: ordered lock-diagonal Structure (Schur complement) Φ Φ n n } 1 n {{ Φ 0 } Φ L 1... = L n } L 1,0 L n,0 {{ L 0 } L D 1... } {{ D 0 } D Cholesky-like factors can be obtained by Schur-complement: L i,0 = i L i Φ i = L i D i L i C = Φ 0 n i=1 L i,0d i L i,0 C = L 0 D 0 L 0 nd the system Φx = b can be solved by L 1 L 1,0.... L n L n,0 L 0 } {{ } L Di 1, i = 1,... n z i = L 1 i b i z 0 = L 1 0 (b 0 L i,0 z i ) y i = Di 1 z i x 0 = L 0 y 0 x i = L i (y i L i,0x 0 ) Operations (Cholesky, Solve, Product) are only performed on sub-blocks Can also exploit structure in sub-blocks urora 2007, Vienna 14

17 Tree Representation of Problem Structure Primal lock ngular Structure Dual lock ngular Structure Primal lock ngular Structure D 1 D 2 D 11 D 12 D D D D D C 31 C 32 D 30 Organisation of Linear lgebra, Parallelism urora 2007, Vienna 15

18 OOPS: Object-oriented linear algebra implementation Every node in block elimination tree has own linear algebra implementation (depending on its type) Implementation is realisation of an abstract linear algebra interface. Different implementations for different structures are available. Matrix Interface matrix factorize SolveL SolveLt y=mx y=mtx orderedlockdiagonal D i Implementations based on Schur complement i SparseMatrix general sparse linear algebra RankCorrector Rank corrector implementations D R Rebuild block elimination tree with matrix interface structures urora 2007, Vienna 16

19 Sources of Structure I (linear) dynamics: x t+1 = x t + u t I I uncertainty: common 1st stage decision (today) recourse action in 2nd stage (tomorrow) T x + W i y i = h i T T T W W W almost independent divisions share a common resource k i x i = b i=1 nd many others.... lso nested structures. urora 2007, Vienna 17

20 LM: Structure of matrices and : Matrix Matrix 1 d d d d 1 urora 2007, Vienna 18

21 Structures of and imply structure of Φ : d d d d 1 1 T T ( 0 T T T T ) T T T T T T T d T d T d T d T 1 d T T d T T P T T T T d ( 0 ) T T d P 1 T T d T d T d T d T T 1 1 urora 2007, Vienna 19

22 Results (LM: Mean-Variance P formulation): Problem Stages lk ssets Scenarios Constraints Variables iter time procs machine LM SunFire 15K LM LM LM UNS LM LM LM urora 2007, Vienna 20

23 HPCx (Daresbury, England) 1600 IM Power-4 Processors 1.7GHz, 800Mb R max = 6.2 TFlops #100 in top500.org list luegene/l (Edinburgh, Scotland) 2048 Processors 0.7GHz, 256Mb R max = 4.7 TFlops #148 in top500.org list urora 2007, Vienna 21

24 LM: Size of largest problem Optimization of 21 assets (stock market indices) Scenario tree needs to capture: correlations of 21 assets correlations over time periods 100 branches over 6 stages = scenarios etter: Scenario tree geometry: million scenarios. Scenario tree generated using simulated geometric brownian motion with mean reversion billion variables, 353 million constraints urora 2007, Vienna 22

25 Issues for Massively Parallel Implementation Communications Memory/Data management Granularity of Computations Sparsity of multilevel linear algebra ootstrapping How to get the data onto the processors urora 2007, Vienna 23

26 Example: ordered lock-diagonal Structure (Schur complement) Φ Φ n n } 1 n {{ Φ 0 } Φ L 1... = L n } L 1,0 L n,0 {{ L 0 } L D 1... } {{ D 0 } D Cholesky-like factors can be obtained by Schur-complement: L i,0 = i L i Φ i = L i D i L i C = Φ 0 n i=1 L i,0d i L i,0 C = L 0 D 0 L 0 nd the system Φx = b can be solved by L 1 L 1,0.... L n L n,0 L 0 } {{ } L Di 1, i = 1,... n z i = L 1 i b i z 0 = L 1 0 (b 0 L i,0 z i ) y i = Di 1 z i x 0 = L 0 y 0 x i = L i (y i L i,0x 0 ) Operations (Cholesky, Solve, Product) are only performed on sub-blocks Can also exploit structure in sub-blocks urora 2007, Vienna 24

27 Exploiting Parallelism: ordered lock-diagonal Structure: Distribution of computations: proc 1 proc n Φ 1= L D L C = L D L Φ = L D L n T n n T n 1 1 T T C = L D L n T 1 T n n n n n C = Φ 0 Σ ic i T 0 C = L D L 0 0 Factorize Solve proc 1 proc n 1 1 z = L b z n= L n bn 1 1 T 1 l = L D z 1 1 T l n= nl n D nzn l = b 0 Σi l i z = L l y = D z n y n= D n z y = D z T 0 0 x 0 = L y T T x = L (y L D x ) T T x n= L n (y n nl n D x ) n 0 Storage: Φ 1 T 1 x1 processor 1 On separate processors On all processors Φ n T n x n processor n Communications 1 n Φ 0 x 0 all processors y y y 1 n 0 urora 2007, Vienna 25

28 Memory Management Data for problem requires 2.6G of memory. need to split information between processors To each node in block-elimination tree a set of processors is assigned Linear lgebra is implemented so that processors communicate when needed Distribution of leading matrix blocks among processors implies Distribution of subordinate blocks Distribution of row/column vector contributions urora 2007, Vienna 26

29 Granularity of Computations: Parallel division of the problem In LM problems matrices up to variables can be treated as unstructured sparse matrices on one processor Problem has: 128 first level nodes with variables each second level nodes with variables each. need to decompose problem at second level (with 1280 processors 3 blocks per processor) 128 nodes 30 nodes 30 nodes urora 2007, Vienna 27

30 Sparsity of Linear lgebra (nested ordered lock-diagonal) Φ 1 1 T.... Φ = Φ n n T 1 n Φ 0 Factorize matrix by Schur complement: Factor Φ i = L i D i L T i, Φ = LLT, L = uild C = Φ 0 i (L 1 i i T)T D i (L 1 i i T) Factor C = L c D c L T c L 1... L n L 0,1 L 0,n C Work tends to be dominated by forming V i = L 1 i T i and V T i V i. Keep Vi as sparse as possible. Requires careful exploitation of sparsity when L i itself is available through Schur-complement. urora 2007, Vienna 28

31 Sparsity of Multilevel lgebra Dominant cost is forming Schur complement: n C = Φ 0 Vi T D i V i, where V i = L 1 i=1 Need to solve a system for each column of T i i T i? =? = T Φ i = L i D i Li i T L i V i = T i urora 2007, Vienna 29

32 Sparsity of Multilevel lgebra Dominant cost is forming Schur complement: n C = Φ 0 Vi T D i V i, where V i = L 1 i=1 Need to solve a system for each column of T i i T i = = T Φ i = L i D i Li i T L i V i = T i urora 2007, Vienna 29

33 ootstrapping Problem: Data for the problem is 25 G. We have 512M/node on luegene. How to get the data on the processors? Data can be represented as core: 64 variables, 22 constraints scenario tree (topology, probability, returns for each node): 2.6 G Still does not fit on one processor Scenario tree data (probability, returns) can be generated/read in on each processor as needed Scenario tree topology needs 30 M This is all that OOPS needs to decide on the allocation of blocks to processors urora 2007, Vienna 30

34 ootstrapping II 1. Read problem dimensions, generate scenario tree topology and matrix tree (30M) 2. Call OOPS: Divide matrix tree among processors, set up data structures on each processor 3. Generate/Read in scenario tree data on every processor 4. On every processor: Call-back to generator to fill in SparseMatrix data 5. Start solution process urora 2007, Vienna 31

35 ootstrapping Set up (shape of) Scenario Tree urora 2007, Vienna 32

36 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) urora 2007, Vienna 32

37 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors 1,2,3,4 5,6,7,8 urora 2007, Vienna 32

38 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors 1,2,3,4 1,2 3,4 5,6,7,8 5,6 7,8 urora 2007, Vienna 32

39 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors 1,2,3,4 5,6,7,8 1,2 3,4 5,6 7, urora 2007, Vienna 32

40 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors Divide between processors 1,2,3,4 5,6,7,8 1,2 3,4 5, ,8 7 8 urora 2007, Vienna 32

41 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors Divide between processors Load data on each processor (core + generate scenarios) 1,2,3,4 5,6,7,8 1,2 3,4 5, ,8 7 8 urora 2007, Vienna 32

42 ootstrapping Set up (shape of) Scenario Tree Convert to lgebra Tree (without data) ssign processors Divide between processors Load data on each processor (core + generate scenarios) 1,2,3,4 5,6,7,8 1,2 3,4 5, Solve 7,8 7 8 urora 2007, Vienna 32

43 Results Stages lk ssets Scenarios Constraints Variables iter time procs machine ,831,873 64,159, ,982, G/L ,415,937 96,239, ,469, G/L ,831, ,646, ,443, G/L ,039, ,875,799 1,010,507, HPCx urora 2007, Vienna 33

44 Results Stages lk ssets Scenarios Constraints Variables iter time procs machine ,831,873 64,159, ,982, G/L ,415,937 96,239, ,469, G/L ,831, ,646, ,443, G/L ,039, ,875,799 1,010,507, HPCx Parallel Efficiency/Scaling nodes peak Mem time Comm Cholesky Solves MatVectProd M 2587 (1.00) (1.00) 956 (1.00) 28.8 (1.00) M 1303 (0.99) (1.00) 485 (0.98) 18.0 (0.80) M 688 (0.94) (0.98) 270 (0.88) 13.0 (0.55) M 348 (0.93) (0.99) 139 (0.86) 9.0 (0.40) M 179 (0.90) 3 93 (0.99) 73 (0.82) 5.8 (0.31) M 94 (0.86) 2 47 (0.98) 39 (0.76) 3.9 (0.23) urora 2007, Vienna 33

45 Further Developments: Warmstarting Possibilities for warmstarting are rarely used in Stochastic Programmming Solution approaches for SP (enders Decomposition/IPM) don t warmstart well Interior Point Methods cannot be warmstarted urora 2007, Vienna 34

46 Results (Efficient Frontier) Exp(X) Mean Variance formulation: max IE(X) ρvar(x) StdVar(X) urora 2007, Vienna 35

47 Results (Efficient Frontier) Exp(X) Mean Variance formulation: max IE(X) ρvar(x) StdVar(X) constraints variables procs ρ = ,321 76, , , ,982,604 16,316, ,575, ,478, %-60% of iterations saved on average urora 2007, Vienna 36

48 Idea: Use Warmstart to speed up solution of Stochastic Program Solve problem on a reduced scenario tree first Copy solution vectors from nearby scenarios to construct starting point for full problem. Solve full problem (using this starting point) Saves up to 60% of iterations on problem up to 4096 scenarios. urora 2007, Vienna 37

49 Construction of the warm-start iterate urora 2007, Vienna 38

50 Construction of the warm-start iterate Nodes in the reduced tree: the solution is already available urora 2007, Vienna 38

51 Construction of the warm-start iterate Nodes in the reduced tree: the solution is already available Remaining nodes: copy the solution from the corresponding reduced-tree node urora 2007, Vienna 38

52 Conclusions: OOPS is a general purpose IPM solver with object-oriented linear algebra Will scale up to massively parallel architecture Can solve problems with 10 9 variables using direct factorization methods llows the solution of realistic financial planning problems Future Work on OOPS: Extend warmstarting procedure: IPM difficult to warmstart Multi-Step Scheme Complexity of such a scheme Carry over to other structures (PDE constrained optimization) Incorporation of iterative solvers (structured pre-conditioners) Integrate into structured modelling language urora 2007, Vienna 39

53 Object-Oriented Parallel Solver (OOPS): References: J. Gondzio and. Grothey, Parallel interior point solver for structured quadratic programs: application to financial planning problems, nnals of OR 152 (2007), Vol 1, pp J. Gondzio and. Grothey, Solving nonlinear portfolio optimization problems with the primal-dual interior point method, European Journal of Operational Research, 181 (2007), Vol 3, p J. Gondzio and. Grothey, Exploiting Structure in Parallel Implementation of Interior Point Methods for Optimization, Tech. Rep. MS , School of Maths, University of Edinburgh, Dec J. Gondzio and. Grothey, Direct Solution of Linear Systems of Size 10 9 rising in Optimization with Interior Point Methods,in: Parallel Processing and pplied Mathematics, Lecture Notes in Computer Sciences urora 2007, Vienna 40