Factors affecting the use of data mining in Mozambique: Constan'no Sotomane 12 November 2013

Transcription

1 Department of Computer and Systems Sciences Factors affecting the use of data mining in Mozambique: Toward a framework to facilitate the use of data mining Constan'no Sotomane 12 November 2013

2 Content Problem Area Research Objectives Research Questions Research strategy and methods Results Conclusions and Future Directions

3 Problem Area High capacity to collect and store data à Rapid increase of the amount of data à huge amount of data; Several data rich project are being established in Mozambique: Government Data Centre, the State Financial Management System (SISTAFE) and the integrated platform of service to the citizen; Data holds important information that can be used to support a better decision making and improve the competitive advantage in organizations;

4 Problem Area (2) Data mining can be used to extracts patterns from these huge amount of data; The implementation of data mining in organization can be affected by several factors dependents to the environment; The implementation of data mining as (an emergent technology) can be expensive and risky; There is the need to determine the factors that influence the application of data mining to avoid failure of the project and lose;

5 Research Objectives To determine the level of awareness and adoption of data mining in Mozambique. To determine factors affecting the use of data mining in Mozambique. The factors will be used to develop a framework to facilitate the implementation of data mining in Mozambique.

6 Research Questions Question 1: What is the level of awareness and adoption of data mining in Mozambique? Question 2: What are the factors affecting the use of data mining in Mozambique? Question 3: How to facilitate the application and use of data mining in Mozambique?

7 Research strategy and methodos (1) Interpretative Exploratory multiple case study Organization 1 à participatory observation Organization 2 à participatory observation Survey Maputo City à Interview and questionnaire Application of data mining using the dataset of Institution 1 STUDY 1 & STUDY4 Exploratory visit to Institutions to determine area for data mining application Selecting two institutions for data mining application Exploratory study on the factorms affecting the use of data mining in Mozabique Participarory- observation Application of data mining using the dataset of institution 2 STUDY 2 Survey in Maputo - use of data mining in Mozambique and related factories STUDY 3 Interview & questionaries Factors Affecting the Use of Data Mining in Mozambican Toward the framework

8 Research strategy and methods (2) Research Ques'ons Q1, Q2 Case Study 1 Organiz. 1 Q1, Q2 Observa'on Study 1,4 Awareness, Factors Case Study 2 Organiz. 2 Q1, Q2 Observa'on Study 2 Awareness, Factors Survey Maputo City Q1, Q2 Interviews Study 3 Awareness, Factors Ques'onnaires

9 Study 1: ICT for Automatic forecasting of Electrical Power Consumption: A case study in Maputo Objective 1: Develop short term forecast model for electricity consumption to complement the manual forecast of electrical consumption currently done at EDM; Objective 2: Observe the use, adoption and potential factors the implementation of the data mining at EDM.

10 The data set Electrical consumption of Maputo city: January 2003 December 2009; Weather data were not available we downloaded from internet (weather underground); Test set: January December 2008 Training set: January 2009 December 2009 Missing values, errors and outliers corrected by replacing with averages; Information about price and procedure were not possible to get

11 Characterization of the Electrical Consumption Using Visualization (1) Long term trend and seasonal variation of electrical load Variation of the electrical load per day and during the 5 years

12 Characterization of the Electrical Consumption Using Visualization (2) The electrical load is strongly correlated with the temperature and follow the season pattern The correlation with humidity is not strong as with temperature The long term trend of electrical consumption has similar pattern with GDP per capita during the period

13 Characterization of the Electrical Consumption Using Visualization (3) The clustering resulted on 3 classes: Class 0 Saturdays, Sundays and Holidays Class 1 Summer Work Days Class 2 Winter Work Days A linear regression Model was used to predict each class of the days L d + 1 ci = b 1 * L d ci + b 2 * T d + 1

14 Some results of the Prediction of Electrical Consumption Real Load Predicted Load Real Load Predicted Load M W H M W H Hours Work days - Summer Hours Work day - Winters Real Load Predicted Load Real Load Predicted Load M W H M W H Hours Saturdays Hours Sundays

15 Nr Observed Aspect/ Factor Finding Coments 1 Awarwness of data mining weak 2 Use of data mining None Forecast of electricity consumption 3 Data issues 3.1 Quality of data Acceptable 3.2 Existence of data warehouse No data werehouse 3.3 data integration Not integrated 3.4 Age of data 9 years available in timeline Available partially, in different granularity, with missing and anomalous values measurement of consumption are keep in different files available in different files, in different formats and from different formats 3.5 Sufficiency of data Acceptable Data of operation, weather not available as required 4 Privacy issues 4.1 Privacy high concern on privacy Avoid to provide certain data 5 Skills and Human resources 5.1 Existence of teamwork Not in data mining 5.2 Existence of skill Not in data mining 5.3 Understanding of data mining and techniques No 5.4 Awareness of data mining No 6 Stakeholder support issues 6.1 Existence of business champion Not in data mining 6.2 Commitment of top management Interested to see the result 6.3 Existence of change management Not observed 6.4 Organizational acceptance the Data mining is not used and is not known 6.5 Use of data mining results The data mining is not used and the awareness among the observed people is weak. It make dificult to observe this aspects 7 Organizational issues 7.1 Functional integration No observed 7.2 Alignment of IT and business Not Observed 7.3 Strategy of outsourcing Not observed 7.4 Interdisciplinary learning Not Observed The data mining is not used and the awareness among the observed people is weak. It make dificult to observe this aspects 8 Financial issues 8.1 costs Not Observed 9 Technology issues 9.1 Existence of adequate software and tools Yes 9.2 Existence of database yes 9.3 Existence of data warehouse No Not specific for data mining It is not relational database, but all the measurement are kept in files

16 Study 2: Extracting Pattern from Socioeconomic Database to Characterize Small Farmers with High and Low Corn Yield in Mozambique: A Data Mining Approach Objective 1: Use of data mining techniques to extract from the socioeconomic database the characteristics of small farmers with high and low maize yields in rural areas of Mozambique; Objective 2: Observe the use, adoption and potential factors the implementation of the data mining at MINAG;

17 The data set We used data of socio economic survey of Ministry of Agriculture of years 2007 and 2008; After cleaning the data were composed by 8821 instances and 109 attributes including attribute class (Corn Yield); Sparse, unbalanced, error, outliers and missing data; Outliers were not treated; Missing data were replaced by? Or new value; Attribute class was split in High Yield (Upper quartile -75%) and Low Yield (low quartile) Hotspot of Weka workbench was used;

18 Results: Tree of association rules for the class of small farmers with High Maize Yield

19 Results: Tree of association rules for the class of small formers with Low Maize Yield

20 Characterization of small formers with Low and High Yield of maize Nr Observed Aspect/ Factor 1 Awareness of data mining 2 Use of data mining 3 Data i ssues 3.1 Quality of data Finding weak None Comments There use of other statistical technique to analyses the dataset Limited quality there are some problem of accuracy, completeness and validity of the data as result of typing or wrong provision of i nformation 3.2 Existence of data warehouse No data warehouse 3.3 data i ntegration Not i ntegrated database Age of data 6 years Sufficiency of data enough data available Privacy i ssues Privacy high concern on privacy Skills and Human resources Existence of teamwork Not i n data mining Existence of skill Not i n data mining Understanding of data mining No and techniques Awareness of data mining No Stakeholder support i ssues Existence of business Not i n data mining champion Commitment of top Not observed management Existence of change Not observed management Organizational acceptance the Data mining i s not used and i s not known Use of data mining results Organizational i ssues Functional i ntegration No observed Alignment of IT and business Strategy of outsourcing Interdisciplinary l earning Financial i ssues costs Technology i ssues Existence of adequate 9.1 software and tools 9.2 Existence of database 9.3 Existence of data warehouse Not Observed Not observed Not Observed Measurement are kept i n a database i n SPSS some data are available from different institutions i n different formats Annual measurements Avoid to provide certain data There are considerable skills on statistics and other data analysis techniques. Limited skill i n data mining The data mining i s not used and the awareness among the observed people i s weak. It make difficult to observe this aspects Not Observed Not observed yes No Aggregated data were available from web and downloadable

21 Study 3: Factors Affecting the Use of Data Mining in Mozambique Objective 1: Determine the level of awareness and adoption of data mining in Mozambique; Objective 2: Determine factors affecting the adoption or use of data mining in Mozambique; Knowing the factors helps to determine when and under which condi'ons it should be implemented

22 Research approach, methodology Mix of qualitative and quantitative research methods Qualitative data : Focus group discussion involving twelve data analysts, ICT experts and managers; Open ended questions from the questionnaire Quantitative data: Closed questions to data annalists, ICT experts and managers; 64 valid from 110 total of respondents from Maputo city The selection of the respondents were the combination of purposive and snowball sampling

23 Result : Factors influencing the use and adoption of data mining Stakeholder support issues Existence of business champion Commitment of top management Existence of change management Non organizational acceptance Not use of data mining User request Organizational issues Functional integration Alignment of IT and business Strategy of outsourcing Interdisciplinary learning Financial issues costs Data issues Quality of data Availability of data warehouse Data integration Age of data (timeline) Insufficient data Skills and human resource issues Existence of teamwork Existence of skill Understanding of data mining and techniques Awareness of data mining Technology issues Existence of adequate software and tools Existence of database Existence of data warehouse

24 Result: Quality of data For the focus group the quality of data in Mozambique is low; The reasons for bad quality of data are: Inadequate data collection process; Historical situation of the country ; Limitation of technology for data collection and storage; Social aspects; Limited human capacity and Non use of data. For the participants of survey the quality of data is not low; The controversy about the perception of the level of quality of data indicates the difference of knowledge about the data: some use data other do not use;

25 Study 4: Short-term Load Forecasting of Electricity Consumption in Maputo Objective 1: After 1 year, verify the progress in term of data quality, availability and integration; Improve the short term load forecast of electrical consumption and apply it to the estimation of DAM of SAPP; Objective 2: verify the progress of the condition for the application of the data mining;

26 The data set Electrical consumption of Maputo city: January 2003 October 2012; Maximum temperature, minimum temperature of Maputo City: January 2003 October 12; Test set: January December 2011 Training set: January 2012 October 2012 Missing values, errors and outliers corrected by replacing with averages; Procedures and prices from the EDM, SAPP, ESKOM bokles

27 Method for forecasting electricity for DAM Df ci t+1 = b 1ci D ci t + b 2ci T M t+1 + b 3ci T m t+1 Working Day Summer Working day Winter Saturday Sunday Holiday Model c1 Model c2 Model c3 Model c4 Model c4 Df c1 t+1 Df c4 t+1 Forested of Electricity consump'on + - D f P k DAM Electricity DAM=round( P k D f ) 0.90 Target data for DM Available Electricity DAM=0 if ( P k D f ) 0 Clustering Expert knowledge (Provided by domain expert)

28 Comparison of the results obtained by model EDM and Model RR. The model RR loss less electricity than the model EDM (1) MAE= 1/N i=1 N Pk (Dr+DAM) M o d e l Model RR EDM MAE

29 Comparison of the results obtained by model EDM and Model RR (2)

30 Comparison of the results obtained by model EDM and Model RR (3)

31 Forecas1ng Electricity for DAM Nr Observed Aspect/ Factor 1 Awareness of data mining 2 Use of data mining 3 Data i ssues Finding weak None Coments 3.1 Quality of data Acceptable Available partially, with missing and anomalous values, quality i s i mproving 3.2 Existence of data warehouse No data warehouse measurement of consumption are keep i n different files 3.3 data i ntegration Not i ntegrated 3.4 Age of data 9 years 3.5 Sufficiency of data Acceptable 4 Privacy i ssues 4.1 Privacy high concern on privacy 5 Skills and Human resources 5.1 Existence of teamwork Not i n data mining 5.2 Existence of skill Not i n data mining Understanding of data mining and 5.3 No techniques 5.4 Awareness of data mining No 6 Stakeholder support i ssues 6.1 Existence of business champion Not i n data mining available i n different files, i n different formats and from different formats available i n timeline Data becoming more available Avoid to provide certain data 6.2 Commitment of top management Interested to see the result The data mining i s not used and the awareness among the observed people i s weak. It make difficult to the Data mining i s not used and i s not observe this aspects known 6.3 Existence of change management Not observed 6.4 Organizational acceptance 6.5 Use of data mining results 7 Organizational i ssues 7.1 Functional i ntegration 7.2 Alignment of IT and business 7.3 Strategy of outsourcing 7.4 Interdisciplinary l earning 8 Financial i ssues 8.1 costs 9 Technology i ssues Existence of adequate software 9.1 and tools No observed Not Observed Not observed Not Observed The data mining i s not used and the awareness among the observed people i s weak. It make difficult to observe this aspects Not Observed Yes Not specific for data mining 9.2 Existence of database yes It i s not relational database, but all the measurement are kept i n files 9.3 Existence of data warehouse No

32 Conclusion & Future direction Low awareness of DM; The application of data mining is at very beginning stage ( Pre-Natal); The identified factors are related to : Stakeholders, organization, quality and integration of data, skills, cost and technology; Develop a capability maturity model to provide guidance to organizations: Benchmarking; Provide means to gauge where there are and were they need to go in term of application of data mining; Provide guideline to move from one stage to a more mature stage.

33 Thank you