Structure and Reasoning in Software Development (A Little Methodology) Michael Jackson The Open University and The University of Newcastle jacksonma@acm.org WG2.3 Working Group on Programming Methodology Lachen, March 1-5 2010 3/18/2010 WG2.3Feb10 1
A little methodology Ordering our thoughts Method consists entirely in properly ordering and arranging the things to which we should pay attention. Descartes, Rules for the Direction of the Mind, Rule V To conduct my thoughts in such order that, by commencing with objects the simplest and easiest to know, I might ascend to the knowledge of the more complex. Decomposing to master complexity Descartes, Discourse on Method of Rightly Conducting the Reason To divide each of the difficulties under examination into as many parts as possible, and as might be necessary for its adequate solution. Descartes, Discourse on Method of Rightly Conducting the Reason This rule of Descartes is of little use as long as the art of dividing remains unexplained... By dividing his problem into unsuitable parts, the inexperienced problem-solver may increase his difficulty." Leibniz, Philosophical Writings 3/18/2010 WG2.3Feb10 2
A little methodology Simplicity and human comprehension I concluded that there are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. C A R Hoare; The Emperor s New Clothes It is very helpful to represent these things in this fashion since nothing enters the mind more readily than geometric figures. Descartes, Rules for the Direction of the Mind, Rule XII 3/18/2010 WG2.3Feb10 3
A little methodology Fragmentation and comprehension Systems as contrivances Decomposition Some criteria of simplicity Recombination Isn t refinement enough? Envoi 3/18/2010 WG2.3Feb10 4
A fragmentation example A puzzle * Alan, Bill, Charlie, Dave, Eddy, Fred, Geoff, Harry, Ian, Joe and Keith are all related. Geoff s uncle s brother is Harry s cousin. Eddy s grandfather is Ian s uncle. Alan is not Fred s nephew. Harry s father is Keith's brother. How many cousins does Geoff have? It s a puzzle! * Some British Sunday newspaper, c 1965 3/18/2010 WG2.3Feb10 5
A comprehensible structure A puzzle How many cousins does Geoff have? It s easy! 3/18/2010 WG2.3Feb10 6
A fragmentation example A London Underground journey planner Display of a simple relation: Each tuple is: (station name, line, segment, id) A segment is a maximal linear stretch (having no fork or join) Station ids are 0,1,2,.. ordered by position within each segment (direction is arbitrary) A typical journey planning question: How can I go from Bond Street to Farringdon? It s a puzzle! Baker Street, Metr n, 1, 17 Baker Street, Bakerloo, 2, 14 Baker Street, Circle, 0, 23 Baker Street, Jubilee, 0, 16 Bank, Central, 0, 29 Bank, Northern, 2, 14 Blackfriars, District, 1, 21 Blackpriars, Circle, 0, 36 Bond Street, Central, 0, 35 Bond Street, Jubilee, 0, 17 CannonSt, District, 1, 23 CannonSt, Circle, 0, 34 ChanceryLn, Central, 0, 31 Covent Gdn, Piccad y, 1, 12 Embankm t, Northern, 1, 19 Embankm t, Bakerloo, 2, 19 Farringdon, Metr n, 1, 21 and 293 other tuples 3/18/2010 WG2.3Feb10 7
A comprehensible structure A London Underground journey planner Harry Beck s description: The lines are shown as wholes (you can trace a journey on the map) Stations are shown where they appear on the lines Journey relationships can be grasped immediately A question: How can I go from Bond Street to Farringdon? It s easy! 3/18/2010 WG2.3Feb10 8
A fragmentation example A program inspired by Carroll Morgan Program Carroll; var p1,p2,p3,p4,p5,p6: integer; while true do { if (p3 > 0) then {p3:=p3-1; p5:=p5-1} else if ((p7 = 0) and (p5 = 0)) then {p1 := p1+1; p2 := p1; p5 := -1; p7 := 1} else if ((p7 = 1) and (p5 = 0) and (p2 = 1)) then {write(p1); p7 := 0} else if ((p7 = 1) and (p5 = 0) and (p2 > 1)) then {writeln(p1, (, p2, ) ); p7 := 0} else if (p5 < 0) then {p2 := p2-1; p5 := p1} else {p3 := p2} } 3/18/2010 WG2.3Feb10 9
A fragmentation example A program inspired by Carroll Morgan Program Carroll; * var p:proc; g,h,i,j,k:integer; procedure p13; begin writeln(g); p:=p2 end; procedure p12; begin writeln(g,'(',h,')'); p:=p2 end; procedure p11; begin if (h=1) then p:=p13 else p:=p12 end; procedure p10; begin k:=k-1; p:=p11 end; procedure p9; begin if (j<>0) then p:=p5 else p:=p10 end; procedure p8; begin i:=i-1; j:=j-1; p:=p7 end; procedure p7; begin if (i<=0) then p:=p9 else p:=p8 end; procedure p6; begin i:=h; p:=p7 end; procedure p5; begin if (j<=0) then p:=p3 else p:=p6 end; procedure p4; begin h:=h-1; j:=g; p:=p5 end; procedure p3; begin if (k<>0) then p:=p2 else p:=p4 end; procedure p2; begin g:=g+1; h:=g; k:=0; p:=p3 end; procedure p1; begin g:=1; p:=p2 end; begin p:=p1; g:=-10; h:=-10; i:=-10; j:=-10; k:=-10; while true do p end. * Some compiler directives have been omitted 3/18/2010 WG2.3Feb10 10
A comprehensible structure A program inspired by Carroll Morgan Program Carroll; * var g,h,i,j,k:integer; begin g:=1; h:=0; i:=1; j:=0; k:=1; while true do begin g:=g+1; h:=g; k:=0; while (k=0) do begin h:=h-1; j:=g; while (j>0) do begin i:=h; while (i>0) do begin i:=i-1; j:=j-1; end; if (j=0) then begin k:=1; if (h=1) then writeln(g) else writeln(g,'(',h,')') end end end end end. * A structured version 3/18/2010 WG2.3Feb10 11
A fragmentation example A requirement * If this side is active and Heading Select mode is not selected, Heading Select mode shall be selected when the HDG switch is pressed on the FCP (providing no higher priority event occurs at the same time). * Quoted by Mats Heimdahl; Let's Not Forget Validation; Position Paper For VSTTE Workshop, Zurich 2005. 3/18/2010 WG2.3Feb10 12
A fragmentation example An event: back of train moves out of a block * * J-R Abrial; Train Systems; in M Butler et al (Eds), Fault-Tolerant Systems, LNCS 4157, 2006 3/18/2010 WG2.3Feb10 13
Fragmentation and comprehension The nature of fragmentation Each individual fragment can be parsed but its purpose is not comprehensible The purpose of a whole fragment collection is in itself even less comprehensible Fragmentation can be very useful to give a simplified and uniform semantics suitable for analysis by a machine But fragmentation must follow comprehension of the structure and purpose of the whole Fragmentation is a useful tool for analysis but not for developing systems or programs 3/18/2010 WG2.3Feb10 14
Systems as contrivances Dependable systems For lifts, library, traffic, pacemakers, banks, radiotherapy machines Systems are complex Features, feature interaction, automation, heterogeneous problem worlds, system interoperation, The problem world is not a formal system Formal descriptions are mere approximations Systems are contrivances (Polanyi) 3/18/2010 WG2.3Feb10 15
Systems as contrivances A contrivance: Huygens & Coster, 1657 Weight Escapement a1 a2 a3 Gear Train Pendulum a5 a4 Hands Time Q1 Q2 Hands Time A simple contrivance Component configuration: parts and how they work together as shown in the diagram Requirement (purpose): hands should indicate elapsed time Operational principle: The falling weight drives (a1) the gear train and (a5) hands, constrained by (a2) the escapement; each pendulum swing (a3) releases one tooth and receives an impulse, so gear train rotation counts swings; swing period is roughly constant (a4) so hands (Q1) count elapsed time (Q2). Bounded context: upright; stable; earth s gravitational field; calm air; non-abrasive ambience; 3/18/2010 WG2.3Feb10 16
Systems as contrivances Contrivances, science and mathematics Weight Escapement a1 a2 a3 Gear Train Pendulum a5 a4 Hands Time Q1 Q2 Hands Time Apply science and mathematics within contrivance structure For a contrivance, structure is primary Science and mathematics applied to contrivances Explain success and failures, analyse, improve, calculate values, identify relevant laws and infeasible contrivances, identify necessary context, Correct gear ratios? friction effects on escapement? Can impulses counteract friction and air resistance? Is weight heavy enough to drive gear train? Why is pendulum swing period nearly constant? 3/18/2010 WG2.3Feb10 17
Systems as contrivances A traffic-control system Lights Controller X3 X1 X2 Crossing Buttons Light Units B4 B5 B6 B3 Road Layout B1 B2 C2 Pedestrians C1 Vehicles & Drivers Q1 Q2 Orderly, Safe Traffic Road Sensors B7 A system: the machine and the world A problem: develop the Lights Controller It s complicated. How can we master it? 3/18/2010 WG2.3Feb10 18
Systems as contrivances A lift-control system Control Machine X1 X2 X3 X4 Lobby Display Buttons Gear B1 B2 B3 Users Q1 Q2 Q3 Q4 Provide Safe & Convenient Service Building Manager A system: the machine and the world A problem: develop the Control machine It s complicated. How can we master it? 3/18/2010 WG2.3Feb10 19
System structures for comprehension Complexity of contrivances Control Machine X1 X2 X3 X4 Lobby Display Buttons Gear B1 B2 B3 Users Q1 Q2 Q3 Q4 Provide Safe & Convenient Service Building Manager Lights Controller Three questions How to decompose? Problems subproblems Simplicity criteria? Unities of a subproblem or contrivance How to recombine the parts? Answer: Don t be impatient wait and see! X3 X1 X2 Crossing Buttons Road Sensors Light Units B4 B5 B6 B7 B3 B2 Road Layout B1 C2 Pedestrians C1 Vehicles & Drivers Q1 Q2 Orderly, Safe Traffic 3/18/2010 WG2.3Feb10 20
Decomposition and recombination Decomposition and recombination Three disciplines of decomposing A into B, C and D A A A D B C B C D?? B??? C D? Embedded Jigsaw Loose (eg procedure call hierarchy) (eg OO, relational schemas, CSP) (eg subproblem decomposition) Embedded and jigsaw decompositions Decompose and recombine in the same development step Loose decomposition Identify and study parts, recombine later (jigsaw or embedded) Crucial advantages of loose decomposition Separates intrinsic from interaction complexity Addresses composition only when components are understood 3/18/2010 WG2.3Feb10 21
System structures for comprehension Requirements decomposition to subproblems Provide lift service as prioritised by the building manager Priority Control Buttons Gear Users Prioritised Service Building Manager Brake on danger when fault found in lift equipment Ensure Safety Gear Brake on Danger Operate lobby display of outstanding requests and lift positions for users information Manage Lobby Display Buttons Gear Users Display & Users Lobby Display 3/18/2010 WG2.3Feb10 22
System structures for comprehension Instrumental decomposition to subproblems Provide lift service as prioritised by the building manager Priority Control Buttons Gear Users Prioritised Service Building Manager Edit service priority rules Provide prioritised lift service Edit Priority Rules Building Manager Priority Rules Edit- Action Eppects Provide Service Buttons Gear Users Specified Service Priority Rules Priority Rules: a local variable of Priority Control machine 3/18/2010 WG2.3Feb10 23
System structures for comprehension Criteria of subproblem simplicity Some unities Of purpose: eg Air traffic control Of problem world roles: eg Text-editing centre Of operation synchrony: eg Accounting periods Of problem world properties: eg Healthy / faulty lift equipment Of operational phase: eg Aircraft taxi, take-off, climb, Of fixed context: eg Book loans vs membership The price of unity Defer treatment of feature interactions Why it s worth it Two sources of subproblem complexity 1. Intrinsic complexity of the part itself 2. Complexity due to interactions with other parts Separate simple subproblems from their interactions 3/18/2010 WG2.3Feb10 24
Problem decomposition and recombination Recombination and subproblem interactions Some interaction concerns Interleaving eg: Edit priority rules vs Service Requirement elaboration eg: Book loans vs member status Requirement conflict eg: Inter-library vs member loan Switching eg: Service to Emergency Action Domain sharing eg: Phone display: camera vs gps vs email These concerns are ubiquitous Premature composition is a major source of difficulty Oblivious and Non-Invasive composition is impossible 3/18/2010 WG2.3Feb10 25
Problem decomposition and recombination A V-structure for [radical] developments Initial Problem Identify Subproblems Subproblem Analysis Understood Subproblems Composition Design Composition Analysis Understood Problem V-structure to analyse and understand system development problems Top-down requirement and instrumental decompositions Bottom-up subproblem recombination Resolving subproblem interactions Recombination may reveal additional subproblems Subproblems are essentially projections of the original problem Projections in several dimensions Space: subsets of problem domains Time: subintervals of complete system execution Context: predicates over problem world state Subproblem machines are not software modules Implementation must exploit a repertoire of transformations 3/18/2010 WG2.3Feb10 26
Isn't refinement enough? Reconciling refinement and system structure Machine X Customer Warehouse H Post Office P C R0(C) Machine X Warehouse H Post Office P R1(P) Machine X Warehouse H R2(H) Machine X R3(X) R3(X) refines R2(H) refines R1(P) refines R0(C) Refinement steps must be justified by problem domain properties Warehouse given properties Post Office given properties Customer given properties 3/18/2010 WG2.3Feb10 27
Isn't refinement enough? Some challenges in refinement 1 Control Machine X1 X2 X3 X4 Lobby Display Buttons Gear B1 B2 B3 Users Q1 Q2 Q3 Q4 Provide Safe & Convenient Service Building Manager Finding pure problem reductions Machine/Warehouse/PostOffice/Customer is oversimplified Understanding properties of a non-formal problem world A top-level abstraction is a result, not a starting-point Avoiding confusion of given with required properties Many formalisms allow no distinction Inducing a humanly comprehensible structure Operations on a state view is a fragmentation 3/18/2010 WG2.3Feb10 28
Isn't refinement enough? Some challenges in refinement 2 Control Machine X1 X2 X3 X4 Lobby Display Buttons Gear B1 B2 B3 Users Q1 Q2 Q3 Q4 Provide Safe & Convenient Service Building Manager Easy to find a top-level abstraction for a formal system The requirement is formal: eg y 2 x < ε but hard to find a top-level abstraction for a non-formal system What s the top-level requirement for the iphone? for a Toyota? The development must enumerate and prove sufficient properties Properties are captured by invariants but it s hard to say what set of properties is sufficient Development needs a map locating sufficient properties 3/18/2010 WG2.3Feb10 29
Envoi Concluding remarks Formal reasoning is essential to developing dependable systems Formal reasoning alone cannot induce an intelligible structure appropriate to a complex reality Structure and purpose must take precedence, giving a framework for formal reasoning in systems Are these ideas, illustrated by real-world systems, applicable to software more generally? 3/18/2010 WG2.3Feb10 30
Some references Some references Software Engineering Michael Jackson; The Name and Nature of Software Engineering; in E Börger and A Cisternino eds, Advances in Software Engineering, Springer verlag LNCS 5316, pages 1-38, 2008. Problem frames Michael Jackson; Problem Frames: Analysing and Structuring Software Development Problems; Addison-Wesley, 2001. Contrivances and science Michael Polanyi; Personal Knowledge: Towards a Post-Critical Philosophy; Routledge and Kegan Paul, 1958 and U Chicago Press, 1974. Problem reduction (aka problem progression) Lucia Rapanotti, Jon G Hall and Zhi Li; Problem reduction: a systematic technique for deriving specifications from requirements; Open University Centre for Research in Computing TR No 2006/02, 2006. Robert Seater, Daniel Jackson and Rohit Gheyi; Requirement Progression in Problem Frames: deriving specifications from requirements; Requirements Engineering 12, 2 pages 77-102, April 2007. 3/18/2010 WG2.3Feb10 31
Thank you 3/18/2010 WG2.3Feb10 32