A numerically adaptive implementation of the simplex method

A numerically adaptive implementation of the simplex method József Smidla, Péter Tar, István Maros Department of Computer Science and Systems Technology University of Pannonia 17th of December 2014. 1 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Outline Introduction 1 Introduction Pannon Optimizer Numerical errors 2 Hilbert matrix Condition number Large condition number aware logic 3 Tests Solving the Hilbert matrix Primary large condition number detector 2 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Pannon Optimizer Introduction Pannon Optimizer Numerical errors Optimizer library: Linear algebraic kernel Auxiliary classes Primal and dual simplex implementation Alternative sub-algorithms Our simplex solver can solve the problems of the NETLIB Next steps: Improve the efficiency Develop a parallel simplex solver Implement more advanced algorithms Develop and test new solutions Solve excessively instable problems 3 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Linear algebraic kernel Pannon Optimizer Numerical errors Notations: Linear programming problem min c T x Ax = b x j 0, j = 1...n Linear algebraic kernel Provides linear algebraic algorithms and data structures: Vector operations (e.g. dot product) FTRAN: α = B 1 a BTRAN: π T = h T B 1 where B is the actual basis 4 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Numerical errors Introduction Pannon Optimizer Numerical errors Floating point numbers: ( 1) s 1.m 1 m 2...m n 2 e Errors s {0,1}: sign m i : i th bit of the mantissa e: exponent Rounding error: A» B, and B 0 A = A + B Cancellation: Given A and B 0, A -B C = A + B Expectation: C = 0 Error: C = ±ε These errors can create a lot of fake nonzeros, lead to wrong results and slow down the solution process. 5 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Relative errors Introduction Pannon Optimizer Numerical errors x and y are floating point numbers, x, y > 0 δ x : relative error of x Relative error of the summation: Relative error of the subtraction: δ x+y = max{δ x,δ y } δ x y = x x y δ x + y x y δ y, where x > y If x y < 1 the relative error is amplified! Relative errors can accumulate. 6 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Hilbert matrix Introduction Hilbert matrix Condition number Large condition number aware logic Hilbert matrix: H n,n, where h i,j = 1 Example: H 4,4 = i+j 1 1 1 2 1 1 2 3 1 1 3 4 1 1 4 5 1 3 1 4 1 5 1 6, i,j = 1,...,n 1 4 1 5 1 6 1 7 We can construct the following LP problem: min 0 H n,n x = b x j 0,j = 1...n, and b i = n j=1 1 i + j 1 7 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Hilbert matrix Condition number Large condition number aware logic Open source solvers and the Hilbert matrix It is clear that if and only if x j = 1,j = 1...n, the solution is optimal We have tested CLP and GLPK Size GLPK Exact GLPK CLP 3 3 x j = 1±3.997 10 15 x j = 1 x j = 1 4 4 x j = 1±8.271 10 13 x j = 1 x j = 1 5 5 x j = 1±1.75 10 11 x j = 1 x j = 1 6 6 x j = 1±2.661 10 10 x j = 1 INFEASIBLE 7 7 x j = 1±1.157 x j = 1 INFEASIBLE 8 8 x j = 1±1.6001 x j = 1±0.201 INFEASIBLE 20 20 x j = 1±6.298 x j = 1±4.124 INFEASIBLE 100 100 0 x j 24.009 x j = 1±21.682 INFEASIBLE We have used CLP ang GLPK as libraries, the models were generated and solved by C++ programs. 8 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Condition number Introduction Hilbert matrix Condition number Large condition number aware logic Measures, how much the output changes if the input changes ( ) κ(a) = A A, where A S 1 0 = V U T, 0 0 ( S 0 and the singular value decomposition of A = U 0 0 ) V T If κ(a) is large, computing A is difficult The condition number of the n*n Hilbert matrix is very large, it grows as ( (1+ ) 2) 4n O n κ(h 10,10 ) 3.536 10 13 κ(h 100,100 ) 2.42 10 148 9 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Exact solver Introduction Hilbert matrix Condition number Large condition number aware logic Idea: use an exact arithmetic solver For example: open source tools, like mpf_class or mpq_class Problem: too slow use adaptive methods! One hour on this planet is 7 years on Earth. Great, we ll wait here until the solver finishes. 10 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Compute the condition number Hilbert matrix Condition number Large condition number aware logic Computing the condition number is difficult Assume, there exists an efficient and reliable algorithm for computing the condition number The condition number of the following matrix is large, but the problem is stable: min 2n j=n+1 x j [I H] = b x j 0,j = 1...2n, and b i = 1 If the starting basis is the unit matrix, columns of H will never enter to the basis if the condition number of the matrix is large, it does not necessary causes troubles 11 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Control logic Introduction Hilbert matrix Condition number Large condition number aware logic Requirements: Recognize the instable bases Adjust the precision if necessary If the problem is stable, the overhead of monitoring the problem must be close to zero Solution: Return to lower precision arithmetic if the basis becomes stable Primary large condition number detector: heuristic, quick algorithm, indicates that the current basis is instable Secondary large condition number detector: slow, but exact sensitivity analysis The primary detector minimizes the frequency of running the secondary detector minimizes the overhead 12 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Hilbert matrix Condition number Large condition number aware logic Primary large condition number detector We can not compute the condition number directly However, we can detect the effect of the large condition number We propose: The input of the classic FTRAN is vector a: B 1 a = α Create the perturbed ā copy of a Use a modified FTRAN, which computes B 1 a = α and B 1 ā = ᾱ The modified FTRAN perturbs every sum during the computation of ᾱ If r = max{ α, ᾱ } min{ α, ᾱ } is greater than a threshold, it means that the condition number is too large primary alarm 13 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Large condition number aware logic Hilbert matrix Condition number Large condition number aware logic The primary detector is executed if an error occurs, for example: fallback to phase-1 If a primary alarm occurs, the algorithm performs primary detector in the following iterations If primary alarms occur in every next iteration and r does not decrease secondary alarm the algorithm terminates If a primary alarm occurs, the algorithm performs a sensitivity analysis If the sensitivity analysis finds that the result is extremely instable secondary alarm If a secondary alarm occurs, the software restarts from the last basis with modified parameters (enabled scaling, switching to LU decomposition, etc.) In the last resort: The software restarts from the last basis with enhanced precision arithmetic 14 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Tests Introduction Tests Solving the Hilbert matrix Primary large condition number detector The tests were performed on the following environment: CPU: Intel(R) Core(TM) i5-3210m CPU @ 2.50GHz Memory: 8 GiB Operating system: Debian 7, 64 bit Tests with the following Open MP types: multiple precision floating point arithmetic: mpf_class 128 bit mpf_class 256 bit mpf_class 512 bit rational number arithmetic: mpq_class The output of the primary detector has been tested, too 15 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Solving the Hilbert matrix: mpf_class Tests Solving the Hilbert matrix Primary large condition number detector 16 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Solving large scale Hilbert matrix Tests Solving the Hilbert matrix Primary large condition number detector time unit: seconds mpq_class: rational number arithmetic is the most precise but the slowest Size mpf_class 128 mpf_class 256 mpf_class 512 mpq_class 80 0.166 0.172 0.193 6.67 90 0.242 0.2428 0.273 9.92 100 0.324 0.3449 0.375 14.2 500 38.83 40.5 45.53 4038.9 1000 315 322.17 368.19? 17 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Solving the Hilbert matrix: solutions Tests Solving the Hilbert matrix Primary large condition number detector Size IEEE 754 mpf_class 128 bit 3 x j = 1±1.044 10 14 x j = 1±2.449 10 40 4 x j = 1±1.517 10 13 x j = 1±2.457 10 40 5 x j = 1±6.395 10 13 x j = 1±2.379 10 40 6 x j = 1±4.369 10 10 x j = 1±1.635 10 39 7 INFEASIBLE x j = 1±2.478 10 40 8 INFEASIBLE x j = 1±2.372 10 40 20 INFEASIBLE x j = 1±6.016 10 25 100 INFEASIBLE x j = 1±8.983 10 24 500 INFEASIBLE x j = 1±3.467 10 22 1000 INFEASIBLE x j = 1±7.340 10 22 18 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Primary large condition number detector Tests Solving the Hilbert matrix Primary large condition number detector Output of the detector: r = max{ α, ᾱ } min{ α, ᾱ } δ = r 1 Problem 25FV47.MPS STOCFOR3.MPS PILOT.MPS MAROS-R7.MPS Value of δ after the last iteration 3.66059e-08 3.07735e-08 9.22276e-06 1.39086e-10 Hilbert 7*7 0.0167145 Hilbert 8*8 0.524724 Hilbert 20*20 2.05845 Hilbert 26*26 5.45188 Hilbert 100*100 0.323612 19 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Conclusions, next steps Tests Solving the Hilbert matrix Primary large condition number detector The primary and secondary detectors can minimize the overhead of the adaptivity Multiple precision arithmetic can be used for the excessively instable problems We have to integrate the enhanced precision arithmetic to the Pannon Optimizer We have to integrate the large condition number recognizer algorithm The large condition number recognizer can be accelerated with low-level optimization (SIMD architecture) Our goal: Implement a solver which runs fast on the stable problems, but recognizes the excessively instable problems Switches to more precise arithmetic, and solves these problems too 20 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method

Tests Solving the Hilbert matrix Primary large condition number detector Thank you for your attention! The research and publication has been supported by the European Union and Hungary and co-financed by the European Social Fund through the project TAMOP-4.2.2.C-11/1/KONV-2012-0004 - National Research Center for Development and Market Introduction of Advanced Information and Communication Technologies. 21 / 21 József Smidla, Péter Tar, István Maros A numerically adaptive implementation of the simplex method