The Application of Heuristics in Engineering John Ypsilantis, PhD Managing Director, Heuristics Australia Pty Ltd
Engineering is a mixture of science, art and black magic
Overview What is a Heuristic? Deep v Shallow Knowledge Intractable Problems
Overview Advantages and Disadvantages Machine Learning Learning of Sequential Information Technologies and Algorithms Examples of Heuristics in Engineering Automation and Decision Support Concluding remarks
What is a Heuristic? A Heuristic is an element of experiential or anecdotal knowledge which, while not necessarily grounded in formal theory nevertheless exhibits significant effectiveness in its application Also known as a Rule of Thumb
Deep and Shallow Knowledge Knowledge = Facts, information, and skills acquired through experience or education; the theoretical or practical understanding of a subject (Oxford Dictionary) Knowledge can be classified as Deep or Shallow Deep Knowledge is based on formal theory (Education) Shallow Knowledge is based on experience
Deep and Shallow Knowledge Deep Knowledge is epistemological, i.e. Based on mathematical theory Based on physical laws Deep Knowledge is therefore explainable, with reference to underlying theory.
Deep and Shallow Knowledge Shallow Knowledge is experiential, i.e., Based on patterns that have been observed over time Based on trial and error Based on shared experience (without necessarily formulating a theoretical basis) As a result, Shallow Knowledge is generally difficult to explain
Deep and Shallow Knowledge: An Example Consider the knowledge required to drive a car When a novice or learner driver starts out, he/she relies heavily on the theory learned during the course of obtaining a licence. A novice driver inherently relies on first principles (Deep Knowledge) when performing specific driving tasks: changing gears, braking, negotiating intersections etc. With experience, these tasks become second nature, i.e., the driver relies on Shallow Knowledge acquired over time, to perform the same driving tasks
Deep and Shallow Knowledge: Examples Function Minimisation Deep (analytical) Shallow (Gradient Descent)
Deep and Shallow Knowledge Shallow Knowledge is often expressed as a collection of heuristics Can be codified as a collection of rules, networks, graphs, synaptic weights
A Paradox All of us use heuristics in our day to day activities Engineers routinely use heuristics in the processes of design, risk assessment and management of projects HOWEVER: there is a general reluctance to apply heuristic techniques in certain areas of engineering, e.g., control WHY? The black magic factor. We generally don t know why heuristics work, so our fear of the unknown often inhibits us from using them in an engineering context.
Engineering is not an exact science
Intractable Problems An Intractable Problem cannot be definitively solved algorithmically within a finite time However, Intractable Problems may be solved in many cases, for practical purposes, in an acceptable time by the application of shallow knowledge techniques (heuristics) Many, but not all intractable engineering problems involve human intervention, e.g., supervisory as opposed to automatic control
Codification of Heuristics If-Then Rule Sets and Inference Engines Forward Chaining Start with antecedents and infer solutions from consequents Extrapolative Backward chaining Start with consequents and determine solutions by matching antecedents Interpolative
Codification of Heuristics Graphs/Networks and Inference by Graph Traversal Examples include Bayesian Networks and Sequence Graphs
Some Advantages of Heuristic Systems Parallel Processing Dynamic update (to a running process) Real-Time (i.e., correctness of the result also depends on the time it is delivered)
Disadvantages of Heuristics Systems Difficulty in eliciting heuristics from SMEs Brittleness Non-determinism (testing and commissioning) Ongoing effort in maintaining the collection of heuristics
Acquisition of Heuristics: Machine Learning Machine Learning techniques can be applied to automate the acquisition of heuristics Primarily addresses knowledge acquisition, code-base maintenance and brittleness issues
Unsupervised Machine Learning Machine Learning System System Inputs Inference System Outputs
Supervised Machine Learning Supervision Machine Learning System System Inputs Inference System Outputs
Inclusion of Time Sequence in Machine Learning Some application domains, (e.g., SCADA alarm processing, voice recognition), include a temporal or sequential dimension in addition to simple patterns Some machine learning systems (Hidden Markov Model Synthesis, Sequence Graph) take sequence into account Sequence can lend significant additional information to the acquired knowledge
Technologies Production systems Forward-chaining rule-based inference systems Rules are evaluated against inputs, best rules are selected State is updated in working memory, rules reapplied Process continues until a single best result ensues Often include backward chaining as well, e.g., for verification or selection of penultimate results
Technologies 4th Generation Programming Languages (AI Languages) (Usually) backward-chaining rule-based inference systems, or pattern manipulation systems Example: PROLOG Inference is conducted as Linear Input Resolution (Proof-by- Contradiction) Inverted hypothesis is presented and disproved Results ensue as side effects to this proof process Programming is essentially rule- and fact-based, at a very abstract level
Machine Learning C4.5/ID3 Induction of decision trees Supervised machine learning (training set needs to be tagged) Principle (heuristic): partition each set recursively, and in such a way that information gain is maximised (entropy is minimised) Creates shallow (i.e., efficient) decision trees
Machine Learning Version space Supervised machine learning algorithm Learns from both examples and counter examples Space of all concepts is bounded above by the General Boundary, and below by the Specific Boundary. Examples refine (further constrain) the General Boundary, counter examples refine the specific boundary - space is progressively pruned Contents of the boundary sets progressively approximate the underlying concept embodied in the examples Both boundaries may converge, in which case the underlying concept is clearly defined by the examples
Machine Learning General Boundary General Boundary General Boundary Version Space Version Space Version Space Specific Boundary Specific Boundary Specific Boundary Initial State Counter Example Example
Machine Learning Bayesean Network Probabilistic Acyclic State Machine Can be induced but usually manually established with the assistance of a SME Can match sequences of events in the presence of noise
Machine Learning b e a d f c Bayesian Network
Machine Learning Hidden Markov model Special case of a Bayesian Network - Linear Bayesian Network Supervised machine learning Induction of a linear sequence of states from tagged training examples Transitions have an associated probability Set of induced sequences are fired probabilistically - management of noise during sequence matching Can statistically predict subsequent states given the sequence of events that has occurred in the past
Machine Learning Sequence Graph Proprietary algorithm Unsupervised machine learning algorithm Has some aspects of a Bayesian Network but is established automatically and without the need for tagged training examples Ability to learn about and unravel multiple concurrent sequences in the presence of noise
Natural and Biological Paradigms Artificial Neural Networks Supervised or (in some cases) unsupervised machine learning systems Efficient in identifying static patterns in the presence of noise Collections of neurone (summation points) with synaptic weights Synaptic weights are tuned as tagged training examples are presented Principle (heuristic): synaptic weights are adjusted on a gradient descent basis (back-propagation)
Natural and Biological Paradigms Genetic Algorithms Supervised machine learning systems, requiring a fitness function (metric to be maximised) Optimisation/pattern recognition/sequence recognition/vector/heuristic search Not prone to being trapped in local minima Principle (heuristic): simulate replication, crossover and mutation (disturbance) of populations of strings On completion, one of the patterns will be close to the optimum (greatest value for fitness function) Analogous to biological genetics - 4 billion years of evolution cannot be wrong
Simulated Annealing Natural and Biological Paradigms Supervised machine learning systems, requiring an objective function (metric to be maximised) Optimisation/pattern recognition/sequence recognition/vector/heuristic search Not prone to being trapped in local minima Principle (heuristic): Stochastic simulation of molecular dynamics as temperature is gradually reduced. Strings are randomly perturbed with progressively falling amplitude in proportion to elapsed time On completion, the best solution will be close to the optimum in regard to the objective function Analogous to crystallising materials
Swarm Algorithm Natural and Biological Paradigms Supervised machine learning systems, requiring an objective function (metric to be maximised) Optimisation/pattern recognition/sequence recognition/vector/heuristic search Not prone to being trapped in local minima Principle (heuristic): Locate the string with greatest objective value, and accelerate all other particles toward this. Acceleration is proportional to the Euclidean distance from this string On completion, the best solution will be close to the optimum in regard to the objective function. Most or all of the strings will have been perturbed to lie close to this optimum Dynamics resemble a swarm of bees or flock of birds converging on prey, food etc.
Applications Intractable or pattern recognition problems, problems relying on sequences, or nonlinear optimisation. For example (power) Fault detection and diagnosis Alarm processing System Security Network Reconfiguration Economic dispatch
Applications The supervision of a utility (power, gas, water, transport) Controls are issued manually by a human operator Automated decision support based on heuristics
Applications (Biomedical) Ensuring that a cardiac pacemaker only operates when it is needed Trigger on ECG pattern Frequency domain analysis and interpretation of ECG using pattern recognition techniques Must operate robustly in the presence of noise
Applications Building Services Security Analysis of normal patterns of access card use Alarming on unusual patterns, possible indication of a lost or stolen card
Applications Application of engineering judgement in the design of a system, structure etc. Application of acquired and expressed design rules Application of standards
Applications Optimisation of nonlinear and time varying objective functions, e.g. Economic Dispatch of generation Heuristic optimisation (swarm, simulated annealing, GA etc)
Managing Deployment of Engineering Systems using Heuristics Design Considerations: Limit the system s level of control, in alignment with risk, e.g., decision support v direct control Include layers of safety and integrity, to cater for brittleness, Design for fault tolerance (separate design teams, voting systems, etc) Incorporate machine learning to facilitate automatic adaptation to the system s environment In short, design on the basis that the system will work effectively most of the time, but will occasionally mis-operate
Managing Deployment of Engineering Systems using Heuristics Pre-Commissioning, Commissioning and Deployment Considerations: Simulate the system for the widest practical set of conditions, Test for brittleness, by simulating inconsistent or unexpected inputs, Give end users sufficient time to gain comfort with and trust in the system, particularly for decision support systems Monitor performance of the system over its lifetime
Conclusion There are problem spaces in Engineering which can benefit from the application of heuristic and machine learning techniques Heuristic and machine learning techniques can (and should!) be considered alongside other techniques in the engineering process Must remain aware of important issues such as brittleness, and manage the engineering process (design, commissioning etc) appropriately
Thank You