Towards the Next Generation Intelligent BPM In the Era of Big Data. Xiang Gao, China Mobile Communications Corporation

Transcription

1 Towards the Next Generation Intelligent BPM In the Era of Big Data Xiang Gao, China Mobile Communications Corporation

2 Outline Background From BPM to ibpm A Big Data Perspective on BPM Embrace the Idea of Big Data Conclusion and Future Work

3 Brief Introduction of CMCC As the leading Chinese telecommunication company, China Mobile Communications Corporation (CMCC) is also recognized as the world's largest mobile phone operator by subscribers with about 740 million. In 2012, the Company was once again selected as one of the "FT Global 500" by Financial Times and "The World's 2,000 Biggest Public Companies" by Forbes magazine. Base stations (thousand) Subscribers (million) Over 1 million base stations covering 99% of national population, and roaming service to 237 countries and regions Over 700 million subscribers, including over 100 million 3G subscribers The world s largest teleco BASS data warehouse over 1200TB The world s largest daily signaling data/billing data over 100/10 TB The world s largest teleco business process repository over processes

4 State of the Art of BPM BPM is always recognized by CMCC as a kind of holistic management approach. By borrowing the basic idea of Gartner's hype cycle, we provide a qualitative graphic representation of the maturity, adoption and social application of specific BPM technologies in CMCC s view. Expectations process fragmentization cloud-enabled BPMS platform process repository business artifacts BPM Suite fragment merging process mining Ontology based business behavior/rules modeling distributed engine BPMS evaluation process models clustering workflow pattern and applications similarity analysis modeling language and mutual conversion More attention has been focused recently on advanced analytical technologies, especially on interdisciplinary collaboration in both research and application. SOA business process analysis Technology Trigger Peak of inflated expectations Trough of disillusionment time Slope of enlightenment Plateau of productivity

5 Great Challenges Arise from IT Consolidation In the context of IT systems consolidation, such a large number of business processes must cause an extremely arduous task for maintaining and evolving. Accordingly, one must elaborately consider business process consolidation first. business understanding & raw process reconstruction complex business logic & recessive rules flexible modeling based on business semantics redundancy removal & process repository During more than 10 years IT construction, a large number of business processes have been varied and updated for many times, and thus, have been biased from the original design. To achieve the objective of consolidation, great attention should be paid to deep understanding of business behavior and efficient process reconstruction. The current processes contain so many special and complex business logics, making the design and execution of processes extremely complicated. Formal design of these business logic should be of much consideration. Actually, business analysts have deep understanding of business but cannot design the process models independently without the support of IT staffs. It is really important to provide an efficient approach to assist flexible and agile design of process models for business analysts with the least IT efforts. Different subsidiary organizations follow unified business specifications and design their own processes. Due to individual management requirements, their processes, even expressing the same business behavior, are usually not exactly the same while having a high degree of similarity. The technology to reduce duplications and make the differences between process models explicit is really important.

6 Example: OA Systems Consolidation Office Automation (OA) is one of the most important management information systems for governments and enterprises in China, to process and communicate information for daily working of all the users. OA fundamentally refers to supporting document flow, approval, transfer, archive and other enterprise general management business processes. The OA systems of CMCC have been independently built for more than 10 years by each subsidiary organizations. more than 3 kinds of OS more than 4 kinds of RAID more than 5 kinds of middleware more than 49 system integrators more than 240 external interfaces more than 1000 application modules more than 8190 business processes It is causing increased architecture heterogeneity, high integration complexity and especially high construction and maintenance cost. These processes are described differently and each subsidiary may have its own description, mainly based on informal user-definition or natural-language text documents.

7 Challenge I: Diversity of OA Different subsidiary organizations follow unified business specifications and design their own processes. Due to individual management requirements, their processes, even expressing the same business behavior, are usually not exactly the same while having a high degree of similarity. Statistics of OA Samples Source: Jan Mendling. Metrics for Models: Empirical Foundations of Verification, Error Prediction, and Guidelines for Correctness, LNBIP 6, Springer, 2008 Let u1, u2, u3 express the normalized mean vectors of different subsidiaries, then the autocorrelation matrix R has the form u1, u1 u1, u2 u1, u3 2, 1 2, 2 2, 3 u, u u, u u, u R R R R R u u R u u R u u R R R T where i j R ui, uj uu u u es belonging to different subsidiaries have high similarity. i j

8 Challenge I: Diversity of OA Different subsidiary organizations follow unified business specifications and design their own processes. Due to individual management requirements, their processes, even expressing the same business behavior, are usually not exactly the same while having a high degree of similarity. Statistics of OA Samples high graph sparsity (edge density=0.07) high control-flow complexity (control-flow complexity = 16.41, when compared with number of tasks) weakly structured (structuredness = 0.61, include 5 arbitrary cycles) Source: Jan Mendling. Metrics for Models: Empirical Foundations of Verification, Error Prediction, and Guidelines for Correctness, LNBIP 6, Springer, 2008

9 Challenge I: Diversity of OA Different subsidiary organizations follow unified business specifications and design their own processes. Due to individual management requirements, their processes, even expressing the same business behavior, are usually not exactly the same while having a high degree of similarity. Statistics of OA Samples Within one subsidiary organization, different processes are highly different. Source: Jan Mendling. Metrics for Models: Empirical Foundations of Verification, Error Prediction, and Guidelines for Correctness, LNBIP 6, Springer, 2008

10 Challenge II: Complex Business Logic of OA The OA processes usually contain plentiful complex business logics, where special attention should be paid to both theoretical and practical issues concerning analysis and realization. registrat ion Approval by Dept. Manager Department There are several difficulties for current BPM technologies in attempts to handle such complex business logics. For example: Carry Out Department Take Dept. COUNTERSIGN fragment for example. It is a common part within most of document approval processes, and can be modeled as parallel multi-instance plus recursive sub-process structure. 1.How to measure the similarity between two of such models? 2.How to mining such models from the corresponding execution log? 3.How to analyze the soundness of such models? 4.How to do indexing and searching on such models? 5.How to implement execution, monitoring and conformance? 6.

11 Challenge III: Big Gap between Business & IT Realization There is always a big gap: business analysts have deep understanding of business but cannot design the process models independently without the support of IT staffs, even though notation based modeling language is exploited. An intelligent configuration tool for process models design is needed to assist business analysts with easy and flexible process models design. Provide business analysts numerous process templates and fragments templates with specific business semantics, automatically extracted from existing process models Provide intelligent process models and log analysis capability, e.g., similarity, search, merging, mining and so forth Provide statistics capability for all kinds of process data template Scenario of Modeling by Business Analysts + Organization s and Role s templates E-Form templates Rules, external services models

12 Outline Background From BPM to ibpm A Big Data Perspective on ibpm Embrace the Idea of Big Data Conclusion and Future Work

13 On the Way from BPM to Intelligent BPM The IT systems consolidation highly depends on the consolidation of business processes first, where intelligent BPM (ibpm) has been given new impetus by integrating analytical technologies into orchestrated processes. Evolution Workflow 1990s BPM 2000 BPMS Mid 2000s ibpm 2012~ meet the ongoing need for process agility, especially for regulatory changes and more-dynamic exception handling aim at leveraging the greater availability of data from inside and outside the enterprise as input into decision making facilitate interactions and collaboration in cross-boundary processes + Mobile Social External Data + = BPMS + Cloud Platform Advanced Analytics ibpms

14 Definition of ibpm from an Industrial Point of View There are a thousand Hamlets in a thousand people s eyes. William Shakespeare Like the famous definition of Big Data by the three Vs, from CMCC s perspective, analytical, automatic, agile and adaptive may also constitute a comprehensive definition of ibpm, and they bust the myth that ibpm is only about analytics. In addition, each of the four As has its own ramifications for analytics. Analytical The most prominent feature of ibpm is the capability of advanced analytics. It integrates with state-of-the-art analytic technologies, including both preanalytics and post-analytics. process model based analysis, such as model decomposition, clone detection, similarity search etc historical log and other information based analysis, such as automatic business process discovery (i.e., process mining), social analysis, intelligent recommendation, prediction etc Automatic The enormous volumes of data require automated or semi-automated analysis techniques to detect patterns, identify anomalies, and extract knowledge. Take process consolidation for example. The ibpm should be designed to facilitate the procedure that automatically reduces duplications and makes the differences between process models explicit, instead of manual operation.

15 Definition of ibpm from an Industrial Point of View BLOG SOCIAL Agile The ibpm is expected to simplify the procedure. For example, by incorporating process fragments with business semantics into design tool, the efficiency of modeling can be significantly improved and most of the procedures can be implemented by business analysts with the least IT efforts. SMART METER Adaptive The dynamic changing of business processes and external data inside and outside should be flexibly captured and responded by resorting to not only the adaptive adjusting of the analysis algorithm parameters, but also the on-demand selection of appropriate algorithms in a configuration way. It is worth noticing that achieving of the 4As features will be given new opportunities in the era of big data.

16 Outline Background From BPM to ibpm A Big Data Perspective on ibpm Embrace the Idea of Big Data Conclusion and Future Work

17 Big Data: Becoming Big Business The birth and growth of big data was the defining characteristic of the 2000s. As obvious and ordinary as this might sound to us today, we are still unraveling the practical and inspirational potential of this new era. Growth of global data Global installed computation to handle information Source: Oracle, 2012 Trends of global data Global installed, optimally compressed, storage Source: Cisco, 2011; Gartner 2009&2011; IDC, 2012 Source: Hilbert and Lopez, The world s technological capacity to store, communicate, and compute information, Science, 2011.

18 Big Data: Advanced Analytical Techniques There is a need for ongoing innovation in techniques that will help individuals and organizations to analyze the growing torrent of big data. A wide variety of technologies has been developed and adapted to aggregate, manipulate, and analyze big data. Association Rule Learning Classification Clustering Crowd Sourcing Optimization Data Fusion Machine Learning Ensemble Learning Genetic Algorithms Neural Networks Pattern Recognition Spatial Analysis Natural Language ing Predictive Modeling Source: Big data: then next frontier for innovation, competition, and productivity, McKinsey Global Institute, June 2011 Network Analysis

19 Big Data: Advanced Analytical Techniques (Deep Insight) The applicability analysis of various algorithms and models have been given new impetus for practice in specific large scale data set scenarios, mainly according to the time complexity and space complexity evaluation. Algorithms with both data intensive and CPU intensive capabilities composite the primary computational mode for big data complicated analysis tasks. Data intensive FP-growth LVM leader K-means CF Hierarchical Model Naïve Bayes SVM Decision Tree Apriori GA For ultra large scale data set and non-real time batch processing, parallel computing is highly recommended after necessary MapReduce type reconstruct of the original algorithm. For real-time task, in-memory computing or stream computing and incremental computing are recommended to increase processing efficiency. Apply probability processing method like Bloom filter\lsh can benefit from time and space complexity decrease. CPU intensive Classification Clustering Association Recommendation

20 Big Data Catalyst for BPM Evolution What does big data really mean in the evolution of BPM? Elegantly stated by the founding father and pioneer long before the introduction of the big data concept. In God we trust; all others must bring data. W. Edwards Deming. Mobile Social External Data Driven by process data and other related data, it can be a new platform for the R & D of intelligence based on big data, making Deming's maxims a reality for the operation of future ibpm systems. Advanced Analytics New methods and tools to embed information into business processes, are making insights more understandable and actionable. Source: Big data, Analytics and the Path From Insights to Value, MIT Sloan Management Review 2011

21 On the Path from Insight to Action (Data) Finding the Needle in the Big Data BPM Haystack The biggest misnomer actually comes from the name itself that is, that big data is about big data. When we talk about big data, we must put its size in relation to the available resources, the question asked, and the kind of data. Traditionally, It is universally acknowledged that Where is business process data? From a special point of view, the complete event log data the process models in centralized repository the process cases data of a large corporation (e.g.,cmcc) can all be treated and analyzed as big data in the BPM field. While, from a generalized point of view, data describing a set of behavior or task with specific order can all be treated as process, such as user clicking on the web, searching Accordingly, a large number of such kind of data are all big data in the BPM field.

22 On the Path from Insight to Action (Analysis) Sparsity Vs. Redundancy The widespread use of traditional data mining and artificial intelligence algorithms has usually exposed their limitations on data sparsity in large-scale data set or problems associated with high dimensionality. However, the large amount of process data always exhibits redundancy instead of sparsity. Identification of highly reusable fragments approval process of province A Vs. approval process of province B reusable fragments (Draft & Approval) For example, user-based collaborative filtering systems have been very successful in the past, but their weakness has been revealed for large, sparse databases Source: Analysis of Recommendation Algorithms for E-Commerce, Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl To deal with more than 8000 OA processes, the technology to reduce duplications and make the differences between process models explicit is really important.

23 On the Path from Insight to Action (Analysis) Sample Vs. Population Sample based analysis is usually conducted to infer the whole behavior of population. However, in the age of big data, one turns to put emphasis on population but not sample, since collecting and processing large amount of data are feasible now. Take the process mining scenario for example, where the completeness of event log plays an extremely important role. For limited event log (also recognized as sample), the global completeness needs to be evaluated by resorting to distribution fitting or at least bound estimation. Vs. For complete event log (also recognized as population), it seems the global completeness is definitely guaranteed. However, the data quality can also affect the efficiency of mining algorithms, while it suffers from data missing and noise infection for population data. Arbitrary Jump Therefore, one must pay more attention to unprecedented challenges when population based analysis was implemented, such as noise cancellation, redundancy removal, data quality improvement and so on.

24 On the Path from Insight to Action (Analysis) Individual Vs. Network Big data is often associated with complex data network, so that it is offering a fresh perspective, rapidly developing into a new network science discipline. It also has exerted a subtle influence on the BPM research and applications world business processes people machines components organizations models analyzes supports/ controls specifies configures implements analyzes software system records events, e.g., messages, transactions, etc. Sample of Cross Enterprise Business Network (process) model discovery conformance enhancement event logs From a business process network point of view, process mining is re-defined as the discipline that covers the discovery, extraction and domain specific analysis of relevant data from dynamic, distributed and heterogeneous enterprise landscapes. Source: From Network Mining to Large Scale Business Networks, Daniel Ritter, WWW 2012 LSNA 12 Workshop, 2012 enterprise people applications machines components models real world, analyzes Business processes middleware network model manages/ controls specifies re-deploys interpretes analyzes discovery conformance enhancement software stack manages,e.g.,c onfiguration,me ssages,process es raw data

25 On the Path from Insight to Action (Analysis) Causality Vs. Correlation A major issue of concern in big data research is that correlation plays much more important role than causality. Google's founding philosophy is that we don't know why this page is better than that one: If the statistics of incoming links say it is, that's good enough. No semantic or causal analysis is required. However, we would like to mention that causality and correlation are equally important in BPM field. mining is strongly based on the rigorous deduction of activity causality from event log. Similarity analysis and clustering based technologies are typical correlation taken into consideration. Clustering Similarity

26 Outline Background From BPM to ibpm A Big Data Perspective on ibpm Embrace the Idea of Big Data Conclusion and Future Work

27 Outline Embrace the Idea of Big Data OA es Consolidation Social Network Analysis Intelligent Recommendation & Matching Test & Application of Distributed Cloud Storage

28 OA Consolidation Procedure I. Mining es with Particular Business Logic Algorithms: improved alpha++ Modeling Language: standard BPMN 2.0 Implementation: tailor made tool (to be shown in the video) business understanding & raw process reconstruction BPMN 2.0 Specification Template Legacy Asset Raw atiza tion Tool mining Tool Temporary Reopsitory Analysis Tool designer models XLS Event Log Raw process level process fragment level Refined process level

29 OA Consolidation Procedure II. ization and Reuse Algorithms: clone detection based on RPST Modeling Language: standard BPMN 2.0 Implementation: tailor made tool (to be shown in the video) BPMN 2.0 Specification VI. Ontology based Rule Modeling Modeling Language: standard BPMN 2.0, OWL Implementation: tailor made tool (to be shown in the video) External Tools: protégé 4.1 Template business understanding & raw process reconstruction Legacy Asset Raw atiza tion Tool Complex business logic & recessive rules mining Tool Temporary Reopsitory Analysis Tool designer models XLS Event Log Raw process level process fragment level Refined process level

30 OA Consolidation Procedure III. Similarity & Clustering Algorithms: clone detection, hierarchical clustering, SSDT-matrix based behavioral similarity Modeling Language: standard BPMN 2.0, Newick format Implementation: tailor made tool (to be shown in the video) External Tools: Legacy Asset Raw BPMN 2.0 Specification Figtree, bpcd atiza tion Tool V. Differentiation based on change operations Modeling Language: standard BPMN 2.0 Implementation: tailor made tool (to be shown in the video) Template business understanding & raw process reconstruction Complex business logic & recessive rules mining Tool Temporary Reopsitory Analysis Tool designer models flexible modeling based on business semantics XLS Event Log Raw process level process fragment level Refined process level

31 OA Consolidation Procedure IV. Merging Algorithms: based on SPL (Software Product Line) Modeling Language: standard BPMN 2.0 Implementation: tailor made tool (to be shown in the video) business understanding & raw process reconstruction BPMN 2.0 Specification Template Legacy Asset Raw atiza tion Tool Complex business logic & recessive rules mining Tool Temporary Reopsitory Analysis Tool designer models flexible modeling based on business semantics XLS Event Log Raw process level process fragment level Refined process level redundancy removal & process repository

32 I. Mining es with Particular Business Logic BPMN 2.0 Specification Template Preemptive dispatch: Multi-instance Parallel Legacy Asset Raw atiza tion Tool mining Tool Temporary Reopsitory Analysis Tool designer models XLS Event Log Raw process level process fragment level Refined process level log parsing OA Log Pattern Recognition Pre-processing Miner preemptive dispatch Combine events Alpha # countersign Events filtering fuzzy Arbitrary Jump Format convert genetic Model Convert BPMN2.O Model Pattern countersign: recursive sub-process 部 门 落 实 Department

33 II. ization and Reuse Legacy Asset Raw mining Tool XLS Event Log BPMN 2.0 Specification atiza tion Tool Temporary Reopsitory Analysis Tool Template designer models Raw process level process fragment level Refined process level Based on clone detection and RPST, a procedure is proposed for process model decomposition, and thus, obtains highly reusable fragments. Decompose process models into fragments that are suitable for reuse. Select fragments that are frequently used. The selection can be made manually by experienced business analysts or automatically by thresholds (e.g., size of fragments), according to specific scenario. Source: Vanhatalo J, Völzer H, Koehler J. The refined process structure tree[m]//business Management. Springer Berlin Heidelberg, 2008: Motivation s reuse can reduce duplications between processes. based process modeling significantly improves the process design efficiency. By testing, the average duration for modeling a process can be reduced by 20% to 60%, when compared to the common approach without fragments with specific business semantic can assist business analysts directly design processes with least IT efforts. ization of sending document process model Uba R, Dumas M, García-Bañuelos L, et al. Clone detection in repositories of business process models[m]//business Management. Springer Berlin Heidelberg, 2011: Gao X., Chen Y., Ding Z., et.al, Model ization, Clustering and Merging: An Empirical Study

34 Calculate SSDT III. Similarity & Clustering BPMN 2.0 Specification Template 2. Newick format Legacy Asset Raw mining Tool atiza tion Tool Temporary Reopsitory Analysis Tool designer models (A:0.1,B:0.2,(C:0.3,D:0.4):0.5); distances and leaf names XLS Event Log Raw process level process fragment level Refined process level 1. SSDT-matrix based behavior Similarity M 1 : approval process of province A 3. Hierarchical Clustering (Agglomerative Clustering) about 0.75 The closer two elements are connected, the more similar they are M 2 :approval process of province B D R AI A P D R AI A P D D 1 2 R 1 3 R 1 AI 2 AI A 1 A P Similarity=1-4/25 P = = 0.84 Source: FigTree 1.4,

35 IV. Merging Legacy Asset Raw mining Tool XLS Event Log Raw process level BPMN 2.0 Specification atiza tion Tool Temporary Reopsitory Analysis Tool Template designer models process fragment level Refined process level Addition: Weaving Configurable Model Blocking and hiding are the essential concepts of configuration. Blocking Hiding Blocking Subtraction Action Activating The merging operation converts a set of similar fragments into one merged fragment (also referred as master fragment), by borrowing some basic ideas from the software product line (SPL). Variation points in the merged diagrams represent locations where the input models disagree in their behavior. That is, a variation point occurs when several alternative flows that belong to different input processes go out from a common activity.

36 V. Differentiation based on Change Operations Legacy Asset Raw mining Tool XLS Event Log Raw process level BPMN 2.0 Specification atiza tion Tool Temporary Reopsitory Analysis Tool Template designer models process fragment level Refined process level Objectives Always get a sound model from a sound model by change operations. The correctness of OA process models/fragments changing and maintenance can be naturally guaranteed More syntactical meanings and much less than change primitives. Make sure that changing operation can be structured recorded. identifying change operations between models Match nodes between two models (one to one match). Compute the (minimal) movements. These movements that are operated on the original model, to match the relationship between matching nodes of models. Model A Model A2 typeset Archive Model B Delete nodes from the original model. These nodes do not have a matching node in the target model. Model A1 typeset Archive Insert nodes into the original model (at the right place). These nodes are in the target model but do not have a matching node in the original model. Model A3=Model B typeset Archive Source: Li C, Reichert M, Wombacher A. On measuring process model similarity based on high-level change operations[m]//conceptual Modeling-ER Springer Berlin Heidelberg, 2008: print

37 VI. Ontology based Rule Modeling Legacy Asset Raw mining Tool XLS Event Log BPMN 2.0 Specification atiza tion Tool Temporary Reopsitory Analysis Tool Template designer models Raw process level process fragment level Refined process level Motivation Business processes are always influenced by legal and regulatory constraints according to managerial requirements. These kinds of constraints are always recessive. Ontology based rule modeling can avoid redundancy and keep consistency Ontology based rule modeling can make information sharable and exchangeable Implementation ontology is designed to allow modeling of external and internal regulations as guidelines and constraints on the interaction between entities and on states of process template. Standard specification of a given language is provided as a descriptive facility and OWL is adopted to describe business rules.

38 Prototype Show In the context of OA consolidation, a specialized configuration tool has been built, which integrates all the aforementioned algorithms as well as tools, aiming at providing flexible and efficient process modeling capability for both the business analysts and IT staffs. Integrative modeling tool User friendly interface Open source based Pluggable modules intelligent analysis Complete statistics A short video Snapshot of the fragment based process configuration tool

39 Embrace the Idea of Big Data OA es Consolidation Social Network Analysis Intelligent Recommendation Test & Application of Distributed Cloud Storage

40 Enterprise Social Network Analysis China Mobile has built a SNA platform based on Big Data frameworks including Hadoop, Graph and No SQL Databases. The platform collects the customer behavior from SINA and TENCENT WEIBO (microblog). We proposed a number of analytic models for social media, such as propagation tracing, TDT, user and group portrait. Enterprise Applications Marketing & Sales Social CRM Brand and Opinion Mgmt. Social Media Operation Big Data Visualization Big Data Visualization Traditional BI Visualization Portal & Integration Analytic Models for Social Media Propagation Tracing Public Topic Detection & Tracing User Portrait Group Portrait Big Data ing & Mining Semantic Entity Analysis Sentiment Analysis Spam Detection Social Search Natural Language Machine Learning & Data Mining Graph Analysis Big Data Collect MapReduce BSP (Hama) Traditional OLAP & Storage APIs & Spider Graph Database No SQL Database MySQL

41 Algorithms in SNA The SNA platform involves quite lots of algorithms: 1) Graph theory, including structure hole, clustering and centrality computation; 2) Text mining, including sentiment analysis, natural language processing and topic detection; 3) links /attributes prediction and inferring, including SVM, normalized conductance and supervised random walking. Topic Detection & Tracking: Latent Dirichlet Allocation LDA is a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model. We have proposed a novel TDT method combining with clustering preprocessing, which improves the precision and recall-rate. Inferring Missing Attributes: Supervised Random Walking Inspired by Prof. Jure Leskovec from Stanford, we implemented a supervised random walking method to infer the missing attributes, such as school and company, of certain user. The supervised random walking method combines the social structure and the known attributes. Therefore we achieve much higher accuracy than ordinary machine learning method. Sentiment Analysis: REN-CECps To evaluate the sentiment of each short-phase microblog, we introduce an eight-emotion analysis method based no REN-CECps dictionary. The emotions are: Expectation, Happiness, Like, Surprise, Anxiety, Sad, Angry and Hate.

42 Enterprise Social Network Analysis (continued) We have gathered about 20 million user and 1.2 billion microblogs. By using our SNA models, we achieved useful tools for enterprise management, such as public opinion tracing, VIP discovery, satisfaction survey and novel CRM. Public Opinion Tracing Mining public topics and opinions from one billion micro-blogs Method: Text Mining, Spam-trim,TDT, Satisfaction survey National-wide, 100 thousand user satisfaction survey Method: Sentiment analysis, NLP Opinion leader and VIP discovery Discovery 244 opinion leader and VIPs out of 100 thousand users. Method: Graph Clustering, Spam detection Novel CRM Based on user s satisfaction and his influence, we draw a user-care map Method: Sentiment analysis, NLP, Social influence

43 Embrace the Idea of Big Data OA es Consolidation Social Network Analysis Intelligent Recommendation Test & Application of Distributed Cloud Storage

44 Background and Targets MM (Mobile Market) of CMCC is similar to Apple App Store, which provides not only more than 150,000 mobile apps, but also a huge number of other digital commodities. Information overload problem of MM is becoming more and more challenging. Truly massive data More than 200,000,000 users More than 150,000 apps More than 1,500,000 audios More than 1,800,000 videos More than 50,000 books More than 60,000 comics More than 100,000,000 PVs per day More than 10,000,000 downloads per day Daily generated log data is up to 50GB now, resulting to a more than 20TB annual accumulation However, the data growth speed is still rising Severe information overload Poor download rate Very hard for users to choose items of their interest Very hard for developers to promote products effectively Targets of Intelligent Recommendation To Realize tripartite win of the users, developers and operators of MM For users: better user experience For developers: equitable opportunity to promote products efficiently For operators: improving operational income

45 System Architecture We built the intelligent recommendation system for MM with open source big data tools, such as Hadoop, MongoDB, Mahout, etc. Data Storage ETL & Modeling Intelligent Recommendation System Architecture Intelligent Rec Engine Hybrid CF ALS-LFM MM Log Sys P-FP Results Optimization MM Business Sys MM UI User Feature & Performance Pluggable recommendation algorithms Support customized algorithm parameter sets for different scenes Three algorithms integrated now: Hybrid- CF(Hybrid Collaborative Filtering) P- FP(Association rule mining) ALS-LFM Run Hadoop Mapreduce on top of MongoDB cluster, achieving better performance Complete all offline computing based on 12 months data in 5 hours Every recommendation costs less than 10ms under concurrent PV up to 3000 per second

46 Deeper Insights into Hybrid-CF Among the 3 algorithms integrated in the system now, Hybrid-CF(Hybrid Collaborative Filtering) is designed and developed by our own research group. To improve the traditional item based CF, in Hybrid-CF we take not only user behavior but also characteristics of both items and users into account. Hybrid-CF Architecture MangoD B User behavior Browse, download, review Calculate basic item preference value Item attribute Category, Subject, etc. User profile Demographic & social information Identify Interest Groups by similarity MangoDB Weighted item rating Generate results Comprehensive use of structured data (user behavior) and unstructured data (item attributes, user profile) Customized parameter sets for different item categories and scenes Mapreduce-Style parallel computing on top of MangoDB

47 Reciprocal Recommendation Traditional recommenders is to provide a USER with recommendations of ITEMS likely to be of interest to the user, such as books, movies, mobile APPs and pharmacy products. Apart from this, we also focus on another important class of recommendations named Reciprocal Recommender, where both the USER and the ITEM models represent people. The two sides have similar standing and both have preferences to be satisfied. Scenario 1 : Online Dating Dealing with SMP(Stable Marriage Problem),to find a stable pair between two sets of elements representing men and women. Matching models such as Gale Shapley algorithm are applied. Scenario 3 : Job-Hunting Motivated by the matching between medical students and hospitals in the US, currently known as NRMP (National Resident Matching Program). A bilateral recommendation approach in matching people and jobs. Scenario 2 : Social Networking Recommending people on social networking sites. To help people to create social and personal connections, to expand friend lists. IBM Beehive using Content-plus-Link model to recommend new colleagues. Scenario 4 : Stable Roommates Known as Roommate Finder, Roommate matching Networks. To help students to find their satisfied roommates. Similar to the stable marriage problem, but differs in that all participants belong to a single pool.

48 Algorithms in Reciprocal Recommenders We divide the Reciprocal recommendation issues into two groups. They are bilateral matching recommendations and multilateral matching ones. We currently have successfully modeled and found solutions using algorithms such as Gale Shapley matching and Exact Cover - Dancing Links X Models. Bilateral Matching : Gale Shapley Matching Model Gale-Shapley Algorithm is applied in Online Dating activities, to find a stable marriage matching pair of man and woman, with full consideration of all members' satisfaction requirements. It can be used in many different scenarios to pair items from two sets. In 2012, Nobel Prize in Economics was awarded. Multilateral Matching : Exact Cover Dancing Links X Model We have successfully established dormitory roommates recommendation system and carried out empirical studies in universities. We treated it as an exact cover problem by using the Dancing Links X algorithm described by Knuth. The model has good universality and replicability, can meet the multi-type multilateral matching problems.

49 Embrace the Idea of Big Data OA es Consolidation Social Network Analysis Intelligent Recommendation Test & Application of Distributed Cloud Storage

50 China Mobile Cloud Benchmarking Framework In open source world, there are various different types of NoSQL Databases and cloud storage systems, in order to distinguish which one has better performance in given scenarios, CMCC developed a novel distributed cloud benchmark framework(cm-cbf), based on open source benchmark and analysis software. Structure: Four layer-architecture (see right) focuses on tuning / benchmarking / surveillance / visualization Framework development is based on widely used open source ( Nmon / YCSB++ & IOzone/ ganglia / rrdtool ) Test framework instruction manage Test visualization manage module Test data generation module Coordinated interface China Mobile Cloud Benchmarking Framework (CM-CBF) Load generator Framework configuration manage Multitask manage Result display Test parameter adjust module Initial Multiframework parameter parameter manage adapter Capability: Self developed adapters support different types of cloud databases & serving systems Evaluation scenarios including scaling / availability / elastic / replication / consistency Parameter Optimization: CM-CBF delivers a group of parameter settings recommendations for different user scenarios Network monitor Operation system monitor Data statistic analysis module Document storage statistic Table storage data access manage HBase adapter MongoDB adapter Cassandra adapter Table storage statistic Test result analysis Distributed document access manage GPFS adapter BigInsights HDFS adapter SWIFT adapter

51 CM-CBF Output and Further Development CM-CBF helps determine the most suitable candidate database for different business, based on business scenarios. Selection Principle : (P1) IF: balanced read and write operations ;THEN: Cassandra or MongoDB; (P2) IF: frequent applications for read operations ;THEN: Cassandra or MongoDB; (P3) IF: large data throughput ;THEN: MongoDB; (P4) IF: frequent insert operation with few applications read ;THEN: HBase; (P5) IF: has a very high scalability ;THEN: HBase; (P6) IF: maintain strong consistency; THEN: SQL DB MongoDB or HBase; (P7) IF: high availability requirements ;THEN: Cassandra (P2P structure) (P8) IF: more flexible data types, complex data structures ;THEN: MongoDB; (P9) IF: native support for MapReduce ;THEN: MongoDB or HBase; (P10) IF: provides strong management and maintenance functions ;THEN: MongoDB or Cassandra Next stage of development for CM-CBF supporting big data data warehouse: benchmark the SQL execution ability of big data data warehouse, Hive / impala supporting cloud management system: Comparison of functional ability for cloud management system Openstack / Cloudstack / OpenNebula

52 Metrics and Evaluation of NoSQL Database CM-CBF aims to test the suitability of cloud storages for different scenarios, in order to benchmark the performance, CM-CBF includes six aspects: basic function test, basic performance test, elastic test, flexibility test, high availability test, consistency test. Test terms Test content MySQL Cassandra Basic function Basic performance Data definition, data operation, control, manage & maintain, function and interface System performance with full-read System performance with full plug-in System performance with upgrade System performance with frequent read System performance with frequent update Mongo DB HBase elastic flexibility High availability System expansion with stable frequent read System expansion with stable frequent update ability of dynamic node extension with online service group influence with failed manage node Group influence with failed data node Group influence with failed router node Group influence with failed distributed node Consistency group node synchronies time, estimate system consistency

53 Benchmarking Results Based on CM-CBF, function, performance, expansion, flexibility and consistency of main distributed document storage and NoSQL Databases are tested. The result proves the effectiveness of open source framework and points out the main aspect they need to focus for realization. reliability performance Distributed document storage have standard read & write interface, conveniently replace traditional storage mode Part of the commercial product have better performance and reliability reliability performance Any open source distributed file systems or NoSQL DB all need more detailed parameter optimization, otherwise which may show quite different performances MongoDB have the most familiar operation interface with SQL and good performance and reliability under most case

54 Outline Background From BPM to ibpm A Big Data Perspective on ibpm Embrace the Idea of Big Data Conclusion and Future Work

55 Conclusions and Future Work The best way to predict the future is to create it. Peter F. Drucker Multi-discipline collaboration social network analysis machine learning semantic web data visualization distributed cloud storage Platform & Commercialization build cloud enabled ibpm platform and provide ondemand analytical service integrate various advanced technologies and tools into process engine and accelerate the evolution to ibpm Open R & D Ecosystem build open and harmony environments for both academia and industry promote and encourage open source tools and prototypes for technology innovation and incubation We sincerely hope to promote the relationship with academia, share our idea, devote ourselves to the advanced researches as well as their realization.

56 Thanks for your attentions