MAIS BI Metrics Requirements Gathering Framework

Transcription

1 MAIS BI Metrics Requirements Gathering Framewrk 1

2 MAIS BI METRICS FRAMEWORK Table f Cntents INTRODUCTION... 3 PURPOSE... 3 OVERVIEW... 3 BI METRIC FRAMEWORK CONCEPTS... 6 MANAGEMENT DIMENSION WHAT WE MANAGE... 7 GOVERNANCE DIMENSION WHY WE MANAGE... 9 MEASUREMENT DIMENSION HOW WE MEASURE PUTTING IT ALL TOGETHER REQUIREMENTS CHECKLIST SUMMARY GLOSSARY OF TERMS REFERENCES

3 Intrductin Purpse The purpse f the MAIS BI Metrics Framewrk is t prvide a cnsistent methdlgy fr BI practitiners when gathering requirements fr a BI prject. The metrics framewrk prvides definitins fr cmmn BI terms, templates fr dcumenting metrics, and techniques fr facilitating requirements gathering sessins. Overview What if the University f Michigan culd track curse enrllments during registratin each term and ntify curse administratrs f the need fr additinal sectins based n pre-defined rules? Better yet, what if we culd predict curse demand in advance? Or imprve student recruitment t yield the highest quality students while making efficient use f recruiting dllars? What if we culd track purchasing patterns and direct purchasers t the best price fr the equivalent prduct? Or culd analyze research funding prtflis and predict the prbability f future funding based n the state f the ecnmy and the federal budget and then direct researchers t surces where they are mst likely t get funded. These are examples f Business Intelligence, which is the prcess f cllecting, structuring, analyzing and leveraging f data t turn int easy-t-understand infrmatin, enabling university leaders t make data-driven decisins. 3

4 Business Intelligence Analytical Impact Hierarchy What s the best that can happen? Optimizatin Cmpetitive Advantage What will happen next? Predictive mdeling What if these trends cntinue? Frecasting Why is this happening? Statistical Analysis Alerts What actins are needed? Query/Drill dwn Where exactly is the prblem? Analytics Why is it happening? Reprting Ad hc reprts Hw many, hw ften, where? What is happening? Standard reprts What Happened? Degree f Intelligence Figure Davenprt, Thmas H., Harris, Jeanne G, Cmpeting n Analytics The New Science f Winning, Bstn: Harvard Business Schl Publishing, 2007, pp. 8 4

5 Example: Curse Trending What steps can be taken t increase the size f the prgram? Optimizatin Cmpetitive Advantage What will this size f the engineering prgram be in 2025? What is the impact f decreased demand fr engineering curses? Why is demand fr engin curses declining? Alerts Query/Drill dwn Frecasting Statistical Analysis Predictive mdeling Ntify when enrllment capacity reaches 90% fr ME 101 Which curses are seeing reductin in enrllment? Analytics Why is it happening? Reprting Ad hc reprts Hw many Engineering students enrlled in physics 101? What is happening? Standard reprts What is UG enrllment by term? Degree f Intelligence Figure Davenprt, Thmas H., Harris, Jeanne G, Cmpeting n Analytics The New Science f Winning, Bstn: Harvard Business Schl Publishing, 2007, pp. 8 5

6 BI Metric Framewrk Cncepts Frm a macr perspective, the metrics framewrk appraches BI prjects like a multidimensinal cube by intersecting the cncepts f Management (What), Gvernance (Why) and Measurement (Hw). Gvernance (Why) Measurement (Hw) Management (What) Management Dimensin: Gvernance Dimensin: Measurement Dimensin: Figure 3 3 Encmpasses the functins that are cmmn t the university Human Resurces, Physical, Student Admin, etc. Fcuses n regulatry alignment, with attentin t cmpliance, risk, accreditatin and perfrmance. Describes the structures (measures, metrics, etc.) thrugh which data is transfrmed int infrmatin. 3 Wells, David L., Requirements Analysis fr Institutinal Intelligence, 2008, p

7 Management Dimensin What we manage The management dimensin prvides the perspective t fcus n a specific functin r prcess. The fllwing are a sample f thse functins that are managed at the University f Michigan. Each functin will likely have mre than ne business prcess assciated with it. Fr example, Student Administratin has business prcesses such as Class Scheduling, Degree Audit and Registratin. Gvernance (Why) Measurement (Hw) Alumni/Develpment Financial Human Resurce Physical Student Administratin Student Life Misc Figure 4 4 * Nte: This preceding list is a sample, nt an exhaustive inventry, f the functins at the University f Michigan. 4 Wells, David L., Requirements Analysis fr Institutinal Intelligence, 2008, p

8 Student Administratin: This functin is assciated with Student Admin activities. Business prcesses include activities such as: Tuitin and Fees Financial Aid Recruiting and Admissins Student Orientatin Class Scheduling Registratin Etc. * Nte: This preceding list is a sample, nt an exhaustive inventry, f the business prcesses within the Student Administratin functin. 8

9 Gvernance Dimensin Why we manage The gvernance dimensin prvides the perspective f the purpse f measurement. Why are we measuring? Mitigating risks (plitical, legal, and financial)? Measuring perfrmance? Cmpliance requirements (gvernment, institutinal)? The fllwing are the subjects f the gvernance dimensin. Risk Cmpliance Accreditatin Perfrmance Alumni/Develpment Financial Human Resurce Physical Student Administratin Student Life Misc Figure Wells, David L., Requirements Analysis fr Institutinal Intelligence, 2008, p

10 Perfrmance: The prcess f assessing prgress tward achieving predetermined gals. Fr example, measuring curse enrllment at the 3 rd week by term ver a five year perid culd expse a pattern f declining enrllment. Accreditatin: A type f quality assurance prcess under which services and peratins f an educatinal institutin r prgram are evaluated by an external bdy t determine if applicable standards are met. Fr example, reprts culd be prduced that demnstrate the requirements fr student curriculum established by the accreditatin rganizatin are being met. Cmpliance: The prcess f cnfrming t a specificatin r plicy, standard r law that has been clearly defined. Fr example, demnstrating thrugh audit reprts that factrs such as ethnicity r gender were nt cnsidered during the admissin prcess. Risk: The prcess f identifying, mnitring and limiting risks, which culd include financial, legal and plitical. Fr example, analyzing retirement estimates fr tenure track faculty culd identify departments wh need t increase the number f tenure track emplyees t fill the vacancy. 10

11 Measurement Dimensin Hw we measure The measurement dimensin prvides the ability t measure a gal. This dimensin has tw sub dimensins. The first is the characteristics t be measured vlume, effectiveness, cst, quality, etc. The secnd is the timing f the measurements leading indicatrs f future behavirs, lagging indicatrs f past behavirs, r current indicatrs f the present situatin. Risk Measures Metrics References Trends Indicatrs Indexes Cmpliance Accreditatin Perfrmance Alumni/Develpment Financial Human Resurce Physical Student Administratin Student Life Misc Figure Wells, David L., Requirements Analysis fr Institutinal Intelligence, 2008, p

12 Measures: A measure is a single data value that is quantitative in nature and assciated with the thing that is quantified. Fr example, 100 is a data value. When given the cntext Number f students enrlled, it becmes a measure. Metrics: A metric is a set f measures that is based upn standard units and has sufficient cntext t prvide infrmatin thrugh srting, gruping, filtering, summarizatin, etc. Number f students enrlled by term, academic level and academic prgram is a metric. References: A reference is a cmparative value that gives meaning t a metric. It is the basis by which a metric can be evaluated as gd r bad. References include threshlds, targets, previus values, etc. The gal f increasing enrllment f freshman engineering students in the fall term by 5% frm the previus year is a target r gal reference. Trends: A trend is a specific type f reference in which a series f metric values are cmpared ver a span f time. Trends illustrate patterns f behavir ver time. A five year dwnward trend in enrllment in the engineering prgram might indicate a need t increase recruiting effrts fr this prgram. Indicatrs: An indicatr is a metric that is used t evaluate perfrmance against gals. A key perfrmance indicatr (KPI) is used t evaluate perfrmance against strategic gals. Engineering Prgram Enrllment might be an indicatr f the verall health f the engineering enrllment prcess. Indexes: An index cmbines a basket f measures acrss a range f activities using a predefined calculatin t generate a value that can be cmpared t a base time perid. 12

13 Putting it all tgether Cmbining all f the dimensins helps t align the functins f the university and the management bjectives. The metrics framewrk helps t define what metrics are imprtant t yu. The management dimensin ensures all functins and business prcesses have been cnsidered. The gvernance dimensin prvides the fcus t identify why the prject is needed. The measurement dimensin prvides the structure t answer tw kinds f questins. What kinds f measures are needed? Fr what time frames are the measures needed? Why Risk Measures Metrics References Trends Indicatrs Indexes Cmpliance Accreditatin Perfrmance Hw Alumni/Develpment Financial Human Resurce Physical Student Administratin Student Life Misc What Figure Wells, David L., Requirements Analysis fr Institutinal Intelligence, 2008, p

14 Requirements Checklist The requirements checklist establishes the bundaries f an effrt by defining all f the things it will prduce. We can use the cncepts f the management dimensin, gvernance dimensin and measurement dimensin as a high level checklist when cllecting high level requirements. The purpse f this checklist is t ensure the right questins are asked when defining the metrics fr an effrt. Instead f fcusing n questins like What metric d yu want n this reprt? questins shuld be brader and shuld encmpass all dimensins f the metrics framewrk. Requirements Checklist What are the bjectives? Why are yu ding this? Increased perfrmance Mitigate financial risk Etc. What are the capabilities? What d yu intend t get ut f the prject? Plan Predict Learn Etc. What functins r prcesses are affected? Wh is affected? Finance Human Resurce Physical Etc. What kinds f measures? Hw can we measure it? Effectiveness Vlume/Cunt Cst Etc. What time frames? Past (Lagging) Present (Psitining) Future (Leading) 14

15 When participating in the interview prcess try t ensure yu ve cnsidered each part f the checklist abve. Fr example, the interview may include questins such as What gal are yu trying t achieve? Hw d these metrics enable yu t make infrmed decisins? Perhaps yu are trying t imprve the enrllment prcess and t understand what factrs are mst influential in the prcess. The answer prvides answers t tw parts f the checklist. Functin Capabilities Learn/Understand Imprve perfrmance Objectives Manage perfrmance Measures Time Frames 15

16 Fllw up questins might be smething like What functins r prcesses are invlved with the enrllment prcess? Wh is affected? Wh can influence the enrllment prcess? These types f questins help determine the functin affected. Functin Student Admin Class Scheduling Enrllment Capabilities Learn/Understand Imprve perfrmance Objectives Manage perfrmance Measures Time Frames 16

17 The final questins shuld fcus n hw are we ging t measure? What are thse metrics that can be used t imprve perfrmance r help us understand which factrs are mst influential in the prcess? What are the time frames assciated with the metrics? Typically, leading time frames are used fr predictive/frecasting types f activities, present time frames are used fr mnitring/managing and lagging time frames are used fr understanding/learning. Functin Student Administratin Recruiting and Admissins Capabilities Learn/Understand Imprve perfrmance Objectives Manage perfrmance Measures Cunt f Enrllments Time Frames Present (Psitining) Past (Lagging) During this time, additinal questins begin t arise. What cntext d these metrics need? Hw frequently des this data need t be updated? Hw sensitive is this data? Wh will wn this metric? In ur example, ne f the metrics deemed imprtant is the Cunt f Enrllments. T add cntext, it might be helpful t see this metric by different dimensins such as Academic Level, Academic Prgram, term, etc. It may als be useful t understand this in relatin t time (date) s we can identify patterns in the data. If we are using this metric t manage the enrllment prcess, then we wuld prbably need a daily refresh f the metric. Understanding the enrllment prcess histrically may require a less frequent update schedule. Wh will be respnsible fr this metric and answering questin abut it? 17

18 T capture this type f infrmatin, we can use the fllwing template when defining metrics. Name f the Metric Cunt f Enrllments Descriptin A cunt f individuals wh have enrlled fr curses at the University f Michigan. Frmula Cunt f EMPLID where enrlled status is true. * Additinal transfrmatin rules wuld be included. Prcesses Affected Student Administratin Enrllment Class Scheduling Purpse / Capabilities Manage Perfrmance Plan Understand trends Cntext / Dimensinal Data Term Citizenship Academic Prgram Academic Level Refresh Rate Daily Metric Owner Office f Undergraduate Admissins 18

19 KPI s versus Metrics (Eckersn, 2009) The difference between a metric and a KPI is that a KPI embdies a strategic bjective and measures perfrmance against a gal. The gals attached t a KPI are multidimensinal: they have ranges that are encded in sftware, a time frame by which the gals must be achieved, and a benchmark against which the gals are cmpared (See Table 1) Strategy KPIs embdy a strategic bjective Table 1 Targets KPIs measure perfrmance against specific targets. Targets are defined in strategic, planning, r budget sessins and can take different frms (e.g., achievement, reductin, abslute, zer). Ranges Targets have ranges f perfrmance (e.g., abve, n, r belw target). Encdings Ranges are encded in sftware, enabling the visual display f perfrmance (e.g., green, yellw, red). Encdings can be based n percentages r mre cmplex rules. Time frames Targets are assigned time frames by which they must be accmplished. A time frame is ften divided int smaller intervals t prvide milepsts f perfrmance alng the way. Benchmarks Are measured against a baseline r benchmark. The previus year s results ften serve as a benchmark, but arbitrary numbers r external benchmarks may als be used. 19

20 Summary The BI metrics framewrk prvides a cnsistent methdlgy when gathering requirements fr a BI prject. The imprtant cncepts t remember are why are we ding this, wh is affected and what will the slutin prvide? Keeping these key cncepts in mind while gathering requirements will help ensure the slutin prvides the knwledge r infrmatin needed t better manage r understand prcesses and peple at the university. 20

21 Glssary f Terms Term Accreditatin Ad hc reprts Alert Analytics Attribute Balanced screcard Business Intelligence Cmpetitive advantage Cmpliance Custmer Relatinship Management (CRM) Data Dictinary Data Element Data Mapping Data Mart Definitin A type f quality assurance prcess under which services and peratins f an educatinal institutin r prgram are evaluated by an external bdy t determine if applicable standards are met Reprts that allw the users themselves t create specific, custmized queries. The term used t define a machine-t-persn cmmunicatin that is imprtant, urgent and/r time sensitive. The extensive use f data, statistical and quantitative analysis, explanatry and predictive mdels, and fact-based management t drive decisins and actins. Additinal infrmatin included with a dimensin that is nt used in defining the levels f the dimensin. The balanced screcard is a strategic planning and management tl that is used t align business activities t the visin and strategy f the rganizatin, imprve internal and external cmmunicatins, and mnitr. The prcess f cllecting, structuring, analyzing and leveraging f data t turn int easy-t-understand infrmatin, enabling university leaders t make data-driven decisins. Cmpetitive advantage is gained by expliting the unique blend f activities, assets, market cnditins, and relatinships that differentiates an rganizatin frm its cmpetitrs. The prcess f cnfrming t a specificatin r plicy, standard r law that has been clearly defined. CRM is a business apprach that integrates Peple, Prcesses and Technlgy t maximize the relatins f rganizatins with all types f custmers. A catalg f all data elements, cntaining their names, structures and infrmatin abut their usage. The mst elementary unit f data that can be identified and described in a dictinary r repsitry which cannt be subdivided. The prcess f identifying a surce data element fr each data element in the target data warehuse envirnment. A repsitry f data that serves a particular cmmunity f 21

22 Term Definitin knwledge wrkers. Data Mining Data Warehuse Drill Dwn Enterprise Mdeling ETL (Extract, Transfrm and Lad) Frecasting Framewrk Gap Analysis Granularity Hierarchy Indicatr Key Perfrmance Indicatr (KPI) Lateral Thinking Measure Metadata Metric Mdel The prcess f finding hidden patterns and relatinships in the data. A cllectin f data, frm a variety f surces, rganized t prvide useful guidance t an rganizatin's decisin makers. The prcess f finding mre detailed data by displaying data at a lwer level than was previusly shwn. The develpment f a cmmn cnsistent view and understanding f data elements and their relatinships acrss the enterprise. The prcess f getting data ut f ne data stre (Extract), mdifying it (Transfrm) and inserting it int anther data stre (Lad). The prcess f estimatin in unknwn situatins. A basic cnceptual structure used t slve r address cmplex issues. This very brad definitin has allwed the term t be used as a buzzwrd, especially in a sftware cntext. The study f whether available business data supprts business requirements. The level f detail f the facts stred in a data warehuse. Organizatin f data int a lgical tree structure. A metric that is used t evaluate perfrmance against gals. A measure f smething that is strategically imprtant t the business against a gal 1. A prblem slving methd where yu attempt t slve a prblem by lking at the prblem frm many angles instead f tackling it head n in rder t create and identify new cncepts and ideas. A single data value that is quantitative in nature and assciated with the thing that is quantified. Data that describes that data in the data warehuse. A set f measures that is based upn standard units and has sufficient cntext t prvide infrmatin thrugh srting, gruping, filtering, summarizatin, etc. A mdel in science is a physical, mathematical, r lgical representatin f a system f entities, phenmena, r prcesses. Basically a mdel is a simplified abstract view f the cmplex 22

23 Term Definitin reality. Predictive Mdel Quantitative Data Query References Operatinal BI Slice and Dice Standard Reprt Statistical Analysis Strategic BI Tactical BI Task Frce Trends The prcess by which a mdel is created r chsen t try t best predict the prbability f an utcme. Als knwn as numerical data; is data measured r identified n a numerical scale. In terms f databases, it is the methd t ask questins f the database. The tw mst utilized methds t extract data are SQL (Structured Query Language) and MDX (Multi Dimensinal expressins). A cmparative value that gives meaning t a metric. It is the basis by which a metric can be evaluated as gd r bad. References include threshlds, targets, previus values, etc. Delivers infrmatin t the pint f business - the frnt lines f a business where infrmatin is used as part f an peratinal prcess. A term used t describe a functin at the cre f multidimensinal analysis. Multidimensinal tls allw users t view data frm any angle. A dcument characterized by infrmatin r ther cntent reflective f inquiry r investigatin, which is tailred t the cntext f a given situatin and audience. The prcess f cllectin, examinatin, summarizatin, manipulatin, and interpretatin f quantitative data t discver its underlying causes, patterns, relatinships, and trends. Prvides perfrmance metrics t management and executives, ften in cnjunctin with a frmal management methdlgy such as Balanced Screcard r Six Sigma. Is the applicatin f business intelligence tls t analyze trends, frequently cmparing a specific metric (such as sales r expenses) t the same metric frm a previus mnth r year. A task frce (TF) is a temprary unit r frmatin established t wrk n a single defined task r activity. Originally intrduced by the United States Navy, the term has nw caught n fr general usage and is a standard part f NATO terminlgy. Many nnmilitary rganizatins nw create "task frces" r task grups fr temprary activities that might have nce been perfrmed by ad hc cmmittees. A specific type f reference in which a series f metric values are 23

24 Term Definitin cmpared ver a span f time. Trends illustrate patterns f behavir ver time. 1 The gals attached with a KPI are typically multidimensinal. They generally have ranges that are encded smehw in the reprting tl, a timeframe assciated with a gal is targeted t be achieved and usually have a reference/benchmark t prvide cntext (See Table 1). 24

25 References Cmpeting n Analytics the New Science f Winning, Davenprt & Harris Harvard Business Schl Press, 2007 Requirements Analysis fr Institutinal Intelligence, Wells 25