European Big Data Value Strategic Research & Innovation Agenda

Transcription

1 European Big Data Value cppp - - April 2014 European Big Data Value Strategic Research & Innovation Agenda VERSION 0.7

2 Contents 1. Executive Summary Introduction Vision for Big Data Value Objectives Big Data Value Challenges Innovation spaces as Incubators Research and Innovation for technology and application Skills and Best Practices Ecosystems, Business Models Input to Regulation and Standardisation Work Programmes and Instruments Collaboration Stakeholder Platform Governance Revision index (version 0.7): In version 0.7, section 7 on challenges for research and innovation for technology and application specifies 5 prioritised focus areas but does not specify any internal priority among the 5 focus areas. 2

3 1. Executive Summary Big Data is one of the key assets of the future. Mastering the creation of Value from Big Data will enhance European competitiveness, will result in economic growth and jobs, and will deliver societal benefit. Strategic investments by industry, the public sector and governments, accompanied by forward-looking policies, will enable Europe to take the lead in the global data economy and to reap immense societal benefits from the unique opportunities offered by Big Data Value. In this document, the European Partnership for Big Data Value (EP-BDV) sets out the Strategic Research and Innovation Agenda (SRIA) which must be achieved in order to realise this. The EP-BDV proposes a contractual Public Private Partnership (cppp) to deliver this SRIA within the European research and innovation landscape of Horizon 2020 and national and regional initiatives. For maximum impact, the cppp must build upon, continue and accelerate these initiatives, federating national and European activities, reinforcing and augmenting a Europe-wide research and innovation effort, with clear strategies for exploitation, skills development, and investment to maximise take-up. The SRIA is built upon inputs and analysis from a pan-european multi-stakeholder group of suppliers and service providers from industry and commerce, including large enterprises and SMEs, research and academic institutions, and users of Big Data in many sectors. Discussions and workshops have clearly shown that, alongside vital research and innovation in technologies and applications, many infrastructural, economic, social and legal challenges will have to be addressed in an interdisciplinary fashion. Underpinning successful exploitation will be the availability of skills and access to investment. Therefore European Innovation Spaces and Environments are a central element of the EP-BDV proposals. Offering secure accelerators for experiments with both private data and open data, these will be the hubs for bringing the technology and application developments together, and will be incubators for new businesses and for the development of skills, competence and best practices. The research and innovation priorities identified are, first and foremost, mechanisms for managing privacy and anonymisation, to enable the vast amounts of data which are not open data (and never can be open data) to be part of the Data Value Chain. Alongside these, Europe needs research and innovation in deep analysis, to improve data understanding, in optimized architectures for analytics of data at rest and in motion, in advanced visualization and user experience, and, underpinning these, in data management engineering. A key challenge for Europe is to ensure the availability of highly and rightly skilled Big Data Scientists and Big Data Engineers. In addition to meeting the research, innovation and business challenges set out in this SRIA, education and training will play a pivotal role in creating and capitalizing on Big Data Value. Europe needs strong players along the Big Data Value Chain, and a successful ecosystem will need all of them, with appropriate and often novel business models to maximize their success. Facilitating the development and testing of these ecosystems and business models will be an important role of the European Innovation Spaces and Environments. 3

4 A favourable Regulatory Environment will be paramount to foster take up of Big Data Value technologies and solutions, and the cppp projects will serve to identify challenges and possible solutions. Standardisation is essential to the creation of a Data Economy, and the cppp will support establishing and augmenting both formal and de-facto standards for technologies and for data, notably for interoperability. The Big Data Value cppp will comprise a variety of projects using a variety of instruments, including Research and Innovation Actions, Innovation Actions and Coordination and Support Actions. Lighthouse Projects will create impact, high visibility and awareness, driving towards faster creation and take-up of Big Data Value. European Innovation Spaces and Environments will be enablers for the Lighthouse Projects and will maximise effective collaboration across sectors. Programme-wide collaboration and governance will support the creation of a self-reinforcing set of coherent activities and a harmonized Europe-wide community of practice. We envision a future in which Europe is the world-leader in the creation of Big Data Value. The SRIA set out in this document, delivered by the proposed cppp, will lead to a comprehensive eco-system for achieving and sustaining this, and delivering maximum economic and societal benefit. 2. Introduction The continuous and tremendous growth of data volume, the better accessibility of data, and the availability of powerful IT systems have led to intensified activities around Big Data. Companies and open source communities are developing powerful tools to collect, store, process, analyse, and visualize huge amounts of data. Open data initiatives have been launched to provide broad access to data from public sector and sciences. The resulting economic value of Big Data is also shown in many market studies e.g. one forecast expects the global Big Data market to exceed $47 billion by 2017, which translates to a 31% Compound Annual Growth Rate (CAGR) over the five year period However, it appears that Europe does not play the significant role it could. With a view on the most successful Big Data vendors world-wide, only very few are European. Also European scientific organisations are slow in getting involved in Big Data-related research and European ICT companies are claimed to have a backlog of about 1-2 years with regards to Big Data adoption. Although there are strong reasons for Europe to take advantage of the Big Data market development, there are seemingly barriers that block the creation of strong data driven ecosystems that foster the uptake of Big Data Value opportunities for Europe. In order to address the potential of Big Data for Europe, a significant and wideranging stakeholder group has been formed in order to bring together all interested parties from the Big Data Value chain, ranging from Technologists to Researchers, from the Content Domain to the Environment Domain. Discussions with these stakeholders have sharpened the focus and helped identify concrete objectives for an overall European Big Data Value partnership to stimulate and coordinate activities across research and innovation, policy and regulation, and industry and society. These stakeholders propose a partnership in the form of a contractual Public Private Partnership (cppp). 1 Jeff Kelly et al: Big Data Vendor Revenue and market Forecast (2013) 4

5 The value that the intelligent use of Big Data can generate has been recognised by many private and public organisations. Many national private and public initiatives have been set up or are in the process of being established. There are, for example, national initiatives in Germany 2, France 3, Ireland 4 and the UK 5. Due to the importance of the topic more initiatives will follow in the next years. The value the cppp will add will be brought about by its lifting of national/regional/municipality activities onto European level, connecting and establishing knowledge sharing, and collaborating to advance the technology to support Europe, and ultimately facilitate Europe to have a leading position in the global data market. This Big Data Value cppp will be a demand driven endeavour, where public and private sides have already invested considerable organisational and financial resources. Multiple domain workshops have been organised in sectors ranging, for example, from SMEs, Content and Media, Manufacturing, Health with a thrust to generate a SWOT analysis with issues and opportunities which can address challenges and contribute to swift take-up by industry and impact consumers. The European Union s Horizon 2020 programme is an innovation driven top-down programme that will create fast and significant socio-economic impact and is a key platform to launch and establish the cppp. However, the end user and end user organisations only take up solutions and apply technology if they truly increase value for them in a bottom-up approach. Thus, the key is to maximise these approaches to deliver on the cppp objectives adding the value for European consumers and industry. This (SRIA) describes the need for such a cppp in more detail. It starts with outlining a vision for Europe in 2020 with Big Data Value as the major driver of the European digital economy in exemplar domains. Based on that, this describes the main objective of the cppp. Analysed challenges for the advancement of Big Data Value creation in Europe are then presented in detail according to four categories: innovations spaces, research and innovation, skills and business models. Some of these challenges determine further challenges and inputs for regulation and standardization processes. Derived from the challenges, this SRIA presents a possible work programme and the instruments to achieve it in order to implement the cppp. For example, Lighthouse projects are highlighted as a new instrument to demonstrate the advanced benefits of the cppp. When the cppp is initiated, it will gather a large body of knowledge for the new European data economy with regards to Big Data Value creation, especially with profound knowledge about the immense amounts of state of the art technology applied in many sectors. This knowledge will be effectively absorbed and coordinated and the the cppp will add enormous value also by lifting national/regional/municipality activities onto European level. Only by connecting and establishing knowledge

6 sharing, and by collaboration and advancing the technology, Europe can establish a leading position on the global Big Data Value market. 3. A Vision for Big Data Value Europe in 2020 Europe and the world will significantly change by Smart applications of ICT such as smart grids, smart logistics, smart factories, and smart cities will be widely deployed across the continent and beyond. Ubiquitous broadband access, mobile technology, social media and the generalised use of machine-to-machine (M2M) communication have contributed to the explosion of generated data such that in 2020, the exponential growth of digital data has reached a global amount of exabytes 6. It is estimated that a significant part of this data will include valuable information. Extracting this information and using it in intelligent ways will revolutionize decision making in businesses, science, and society enhancing companies competitiveness and leading to new industries, jobs and services. These expectations and figures are not futuristic predictions; even short-medium term evolution clearly shows this trend in Big Data Value. According to a report from Research and Markets, the Big Data market in Europe is estimated to grow at an annual rate of 32 % over the period Considering this growth in the data value economy, it is expected that an appropriate strategy for supporting research and innovation in Europe will directly imply the creation of a large number IT jobs in the next 5 years in the EU. In terms of economic impact, it has been estimated that the value of Big Data in 2016 in the German market will reach around 1.7 billion euro 8, which implies an approximate value of 20 billion euro for the EU in 2020 (assuming a conservative CAGR of 20%). Big Data impacting and Economic sectors By 2020, European research and innovation efforts will have led to advanced technologies which make it significantly easier to use Big Data across sectors, borders and languages. Innovation will allow extending the capability to use and exploit Big Data to other economic sectors in Europe therefore maximising the crosssector value of data. The following are examples of those sectors that are especially promising with regard to Big Data Value. Healthcare: Applications range from comparative effectiveness research to the next generation of clinical decision support systems which make use of comprehensive heterogeneous health data sets as well as advanced analytics of clinical operations. Of particular importance are patient involvement, privacy and ethics. Mobility, transport and logistics: Urban multimodal transportation is one of the most complex and rewarding Big Data settings in the logistics sector. In addition to sensor data from infrastructure, vast amounts of mobility and social data are generated by smart phones, C2x technology (communication among and between 6 IDC report: THE DIGITAL UNIVERSE IN 2020: Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East, December Big Data Market in Europe, , Research and Markets, University of Texas: Measuring the Business Impacts of Effective Data (2011) 6

7 vehicles) and end users with location-based services and maps. Big Data will open up opportunities for innovative ways of monitoring, controlling and managing logistics business processes. Deliveries could be adapted based on predictive monitoring, using data from stores, semantic product memories, internet forums, and weather forecasts, leading to both economic and environmental savings. Energy: The digitization of the energy system from production, to distribution, to smart meters at the consumer enables the acquisition of real-time, high-resolution data. Coupled with other data sources, such as weather data, usage patterns and market data, accompanied with advanced analytics, efficiency levels could be increased immensely. Existing grid capacities could be better utilized and renewable energy resources could be better integrated. Public Sector: Big Data Value will contribute to increase efficiency in public administrations processes. The continuous collection and exploitation of real-time data from people, devices and objects will be the basis for smart cities, where people, places and administrations get connected through novel ICT services and networks. In the physical and the cyber-domain, security will be significantly enhanced with Big Data techniques; visual analytics approaches can be used to allow algorithms and humans to cooperate. From financial fraud to public security, Big Data will contribute to establish a framework that enables a safe and secure digital economy. Media and Content: By employing Big Data analysis and visualization techniques, it will be possible to allow users to interact with the data, and have dynamic access to new data as they appear in the relevant repositories. Users would also be able to register and provide their own data or annotations to existing data. The environment will move from, few state-orientated broadcasters to a prosumer approach and where data and content is linked together blurring the lines between data sources and modes of viewing. Content and information will find organisations and consumers rather than vice versa and overall the impact will be a seamless content experience. Manufacturing and production: With industry s growing investments into smart factories with sensor-equipped machinery that is both intelligent and networked (Internet of Things, Cyber-Physical Systems), the production sectors in 2020 will be one of the major producers of (real-time) data. The application of Big Data to this sector will bring efficiency gains and predictive maintenance. Entirely new business models are expected since the mass production of individualized products becomes possible and for which consumers may have direct access to influence and control it. Retail: Digital services for customers provided by smart systems will be essential for the success in the future retail business. The retail domain will especially be focused on highly efficient and personalized customer assistance services. Retailers are currently confronted with the challenge to meet the demand of a new generation of customers who expect information to be available anytime and anywhere. New intelligent services that make use of Big Data will allow a new level of personalized and high-quality Efficient Consumer Response (ECR). Environment: Better understanding and managing of environmental and geospatial data is of crucial importance. Environmental data helps to understand how our planet and its climate are changing and also addresses the role humans play in these changes. The European Earth observation programme, Copernicus, aims to provide reliable and up-to-date information on how our planet s climate is changing to provide a foundation which will support the creation of sustainable environmental policies. In addition, the EU project Galileo will be a global network of satellites providing precise timing and location information to users on the ground and in the air. The overall intention is to improve the accuracy and availability of location data to the benefit of 7

8 for example the transport and industry sectors as well as Europe's new air-traffic control system. Big Data as a major driver in the European data economy Value creation from Big Data will be a major driver that will transform the 21st century society, and it will have a disruptive influence on social innovation, economic growth and public services provision. New jobs and businesses will be created and the profession of data analyst will become essential to support decisions at all levels. From healthcare to education, from energy to water management, from entertainment to mobility, in all sectors Big Data Value will be a key factor in fuelling innovation, developing new business models, supporting productivity growth, and coping with societal challenges. Europe has the opportunity to create an eco-system that allows a wide and meaningful exploitation of the potential of Big Data Value by the European industry. Defining and implementing a solid medium and long-term research and innovation agenda on Data Value is a first and necessary building block for exploiting the Big Data innovation potential for companies and governments in Europe. Europe must aim high and envision a future in which Europe has assumed the leadership in the exploitation of Big Data and its application to some of the most critical sectors for Europe from a socio-economic perspective. Big data value becomes a crucial tool for overcoming future scarcity of resources (environmental), making a more efficient use of energy or supporting the public healthcare system in Europe. 4. Objectives The goal of the Big Data Value contractual Public Private partnership (cppp) is to increase the amount of productive European economic activities and the number of European jobs that depend on the availability of high quality data assets and the technologies needed to derive value from them. This will be articulated into actual technology and data assets development, means for benchmarking and testing, development of business models, improving the skills of data scientists and domain practitioners, dissemination of best practices, and analysis of societal impact. The Big Data Value cppp will do this by creating European cross-organisational and cross-sector environments where large and small companies alike will find it easy to discover economic opportunities based on data integration and analysis and develop working prototypes to test the viability of actual business development. This does not necessarily mean moving data assets across borders but rather making data analytic activities possible across them (for example by bringing the computation to the data The Big Data Value cppp will make valuable data assets available in environments that simultaneously support the legitimate ownership, privacy and security claims of corporate data owners and their customers, and ease of experimentation for researchers, entrepreneurs and small and large companies. The incentive for asset owners is that they will get access to advanced technologies and have business opportunities revealed to them by other participants. These environments will support various models of value creation discovery: at one end, corporate entities with valuable data assets will be able to specify, business relevant, data challenges for researchers or software developers to tackle; at the 8

9 other end, entrepreneurs and companies with business ideas to try, will be able to solicit the addition and integration of desired data assets from corporate or public sources. The environments themselves will be data driven both at the planning and at the reporting stage. At the planning stage, they will prioritise for inclusion those data assets that can, in conjunction with existing assets, be argued to present the greatest promise for European economic development (taking in full account the international competitive landscape); at the reporting stage, they will provide methodologically sound quantitative evidence on important issues such as increases in performance for core technologies or reduction in costs for business processes. These reports will form the basis for the next cycle of planning. The environments will not duplicate but rather federate and complement and reuse the activities of similar national incubators/environments, existing cppps and other national or European initiatives (unless this is required for methodologically sound benchmarking activities). Complementary activities considered for inclusion will have to stand the test of expected economic activity development: new data assets and technologies will be considered for inclusion to the extent that they can be expected to open new economic activities when added to and interfaced with the assets maintained by regional or national data incubators or existing cppps. An additional task will be ensuring the spreading of best practices. The successive inclusion of data assets in these environments will in turn drive and prioritise the agenda for funding of data integration or data processing technologies. For example, the existence of data assets of homogenous qualities (e.g. geospatial, time series, graphs, imagery ) would reveal the need to optimise the performance of some core technologies (querying, indexing, feature extraction, predictive analytics, visualization ) which requires methodologically sound benchmarking practices to be carried out in appropriate facilities. On top of the core technologies, one could then build business applications that themselves need to be evaluated for usability and fitness of purpose. The objective is to have new technology translated into new opportunities for business, optimising the functioning of existing industries and potentially establishing entirely new business models along novel data value chains. The viability of such models will itself need to be evaluated in realistic environments. Due to the richness of the data they contain, access to a large variety of integrated software tools and expert community interactions, the data environments will provide the perfect setting for effective training of data scientists and domain practitioners and encourage a broader group of interested parties to engage in data activities. These activities will be designed to complement the educational offer of established European institutions. The data environments will also act as a showcase of the business benefits of data (re)use best practices so as to encourage their transfer to any domain where they would also be appropriate (across country or across sectors). While economic development is the principal objective of the proposed cppp, this cannot happen without taking into proper account the legislative requirements pertaining to the treatment of data as well as ethical considerations. In addition, the cppp will create value for society as a whole by systematically supporting the transfer of sophisticated data management practices to domains of societal interest such as health, environment or sustainable development, among others. 9

10 5. Big Data Value Challenges The strategic considerations about the goals of the Big Data Value cppp above identified a number of blocking points and areas where the Big Data Value cppp should focus its actions. In summary, those focus area are: broadening the availability and accessibility of data sources assessing the economic value of data assets developing Big Data technologies and tools as needed to support best datadriven applications and business opportunities along the data value chain developing data-driven applications and business models providing measurable value to the involved players and by that addressing the lack of convincing use cases testing and benchmarking technologies, applications, and business models addressing the lack of skills and expertise addressing the issues related to security and privacy as well as increasing the level of trust into data and data-driven applications Intensive discussions with stakeholders have clearly shown that besides technology and application many infrastructural, economic, social and legal issues will have to be addressed in an interdisciplinary fashion. Especially when it comes to SMEs, the issues of skills and training, reliable legal frameworks, reference applications and access to an ecosystem become central for a fast take-up of the opportunities offered by Big Data Value. Figure 1: Interconnected Challenges of the cppp with the Innovation space in the centre European Innovation Spaces and Environments will be the central element that should allow addressing the Big Data Value challenges in the focus areas and along 10

11 the dimensions depicted in the Figure above with the central need for skills, best practices, legal, policy and infrastructural frameworks and tools across all the dimensions: 1. The European Innovations Spaces / Environments (EIS/E) are the hubs for bringing the technology and application developments together and cater for the development of skills, competence, best practices. These environments will offer existing technologies and tools from industry and open source software initiatives as a basic service as need to tackle the Big Data Value challenges. 2. Data are at the center of the Big Data Value activities. The EIS/E will make accessible those data assets based on industrial, private and open data sources. The EIS/E will be the secure and safe places that will ensure the availability, integrity, and confidentiality of the data sources. 3. The EIS/E will serve as incubators allowing testing and benchmarking of technologies, applications, and business models. This will provide early insights on potential issues and will help to avoid failures in the later stages of commercial deployments. In addition, it can be expected that this activities will provide input for standardization and regulation. 4. Focused activities will develop Big Data technologies and tools as well as data-driven applications. Focus will be put on those technologies and tools which are needed to support best data-driven applications and business opportunities along the data value chain. The applications and business models have to provide measurable value to the involved players and will help to address the lack of convincing use cases. 5. Developing skills and sharing of best practices will be an important task of the EIS/E and their federation and linking with other existing initiatives at European and national level. 6. New Business Models and Ecosystems will be emerging from exposing new technologies and tools to industrial and open data. The EIS/Es will be the playground to test new business model concepts and emerging ecosystems of existing and new players. 7. The EIS/E will also allow to get early insights into the social impact of new technologies and data-driven applications and how they will change the behaviour of individuals and the characteristics of data eco-systems. Outcome and KPIs Activities addressing the identified focus areas have to tackle the related challenges and to contribute to the goals of the Big Data Value cppp. In the following, a number of KPIs are proposed which should help to measure the success of those activities by indicating to which extent the goals have been achieved: Number and scale of European Innovation Spaces and Environments created. (value to be determined) Number of large-scale experiments conducted in EIS/Es involving private data. (value to be determined) Number of use cases supported by Big Data technologies and applications developed in the cppp. (value to be determined) 10 times faster data processing for deep analysis. (value to be determined) 11

12 Number of PhDs awarded in Data Science or Data Engineering resulting from projects of the cppp. (value to be determined) Availability of metrics for measuring the quality and value of data assets. (value to be determined) Number of standards influenced and supported by the cppp. (value to be determined) Number of start-ups created as a result of cppp activities. (value to be determined) Investment made in exploitation of cppp results. (value to be determined) Number of jobs created as a result of cppp activities. (value to be determined) 6. Innovation spaces as Incubators The European Innovations Spaces, also understood as incubators environments, are the main element to assure that research for technology and application is quickly tested and piloted in a context with maximum involvement of those that would ultimately be part of the respective ecosystems and develop new business with it and those that use it. The notion of the innovation Spaces is developed based on many initiatives around in Europe that have resulted in (physical) infrastructures and incubators such as European Framework Programme Labs FUI PPP OIL; FIRE as well as the SDIL (smart data innovation lab, Germany), TERALAB (France), INSIGHT (Ireland), Berlin Big Data Centre (Germany) and Open Data Institute (UK). Those European Innovations Spaces are representing the secure hubs for hosting sensitive private data and linking them to open data and for bringing the technology and application developments together and cater for the development of skills, competence, best practice, maturing tool and most importantly the safe/secure place to make industry/private data available and accessible. The Innovation Spaces will function as sandbox to emulate real live situation and have technology, application, tools, frameworks and platforms piloted for bringing data and domain experts form supplier and service provider and user- organization from industry and public sector together and develop new business models and experiment ne ecosystems. The technology, application, tools, frameworks and platforms can be based on open source software ore proprietary source code. Innovation spaces will be selected and promoted that have that capability to address different sectors. SDIL for example has the objectives to address Manufacturing, Energy, Smart Cities, Health. Following this examples innovation spaces will be federated that would be complementary in respect to sectors, platforms, technology scope or practitioners. Physical and virtual labs are candidates to function as innovation space with the capability to host benchmarking for technology and business model performance. At the same time the innovation spaces will cater wherever possible for observations of social impact and developing input to the legal and regulatory debate. The cppp will federate und broaden the scope of individual innovation space candidates to enable them where not yet existing to provide for the functionalities described above. The cppp will particularly pay attention to the federation for facilitating continuous exchange of best practice. 12

13 Once the cppp has collected best practice the cppp will be in the position to also support the leverage of other funding sources such as structural and regional funds for setting up local or regional innovation spaces. An important provision by the EIS/Es is to facilitate benchmarking so that businesses, in particular startups and SMEs, are able to measure that their approaches, e.g., a product or a service after it been tested and piloted, also works for the real world. Hence, benchmarking is about comparing e.g. a specific product or a specific service with a peer product or a peer service, respectively. The comparison could address measures such as efficiency, effectiveness, cost, quality, return of investment, etc. Benchmarking can be applied to most business processes, functions, etc. but in the context of the cppp the EIS/Es mainly concentrate on four strands: Business (Model) benchmarking: This type of benchmarking may among other aspects focus on process, financial or investor perspective aspects. In many aspects this is about Best Practices. Product / Technical benchmarking: This type of benchmarking is about finding out how a product or technical system match e.g., the performance or operational cost of some other products or technical systems. Service benchmarking: Besides e.g., performance or cost measure, typically for services is the customer or user experience (UX for short) benchmark. The quality of the user centric approach when it comes to services is vital if the service becomes a success or not. Data Set benchmarking: The data sets are at the core of this BigDataValue.EU cppp. Measuring and ensuring data quality, not only on existing data sets, but also on live data streams, is the main concern to this strand. For all the four strands mentioned above the EIS/Es are planned to facilitate the whole or part of benchmarking processes. Since, benchmarking is not a one size fits all and is highly contexts sensitive, the company or party requesting a benchmark need to have a leading role in the process. The following elements will be part of this service provision EIS/Es: Service for assisting in identifying what is to be benchmarked, and how to benchmark it. Service for assisting in identifying businesses with e.g., a peer product or a peer service useful for benchmark measure. Service for assiting in identifying best practices and measures, Service for assisting in running the benchmark. 7. Research and Innovation for technology and application To identify the key technical challenges that need to be addressed to ensure the development of a European Data Value ecosystem, a two-way analysis was conducted. First, we extracted the most important issues perceived as challenges by end users of various economic sectors by performing a needs analysis. Those needs have been identified in 9 sectorial workshops. Second, this needs analysis was mapped onto the main roles of the different players in the data value chain. Lastly, we cross checked the end users needs to available 13

14 Big Data technical solutions to derive the challenges to be addressed in order to satisfy those needs. Before going in the details of this two-way analysis, the main findings derived from the analysis are presented. The current situation and Europe assets Large US IT and Internet companies currently have an unquestionable lead on Big Data infrastructure & storage techniques. It does not seem the most efficient approach to try and overtake or compete with them by simply repeating what they have already achieved, but rather build on top of the commoditized core that they have established. The fields of Big Analytics & Data Visualization (predictive and decision support systems) is much more open. The EU has an undeniable competitive advantage here, thanks to the very high mathematical and computer literacy level of EU engineers and research scientists as well as the solid base of industries which own most of the underlying data assets, unlike the end consumer data sets dominated by consumer facing web companies in the US. This positioning is a major factor of differentiation and a real asset. It is all the more important that the real added value of Big Data, in terms of scientific and technological innovation, as well as business, lies in application of intelligent decisions driven by the data analysis. Most of the supporting tools and storage architectures are now Open Source (Hadoop, Hive, Spark, Shark, HBase, Riak, Titan, etc.), levelling the playing field for tool vendors in this field. To ease the development of Big Analytics and innovative data visualization solutions it is nonetheless vital to ease the access to big data infrastructures (including hardware platforms but also the storage & data processing capabilities) which are pretty costly. But European countries have already understood this issue and some of them have already set up such platforms, precisely to nurture the ecosystem of SMEs in this area and favor the emergence of new business ideas. Such platforms need to be more than just simply technical platforms. They represent innovation spaces where the different players in the Data Value chain, not only Big Data solution providers but also data asset owners, can meet and jointly develop new applications. This is the first step towards the development of a Data Value ecosystem. The cppp will therefore not push for the development of a new technical platform at the European level. It will instead encourage initiatives taking benefits of the existing national innovation spaces and promote the collaboration between these spaces within larger scale projects. The fundamental, strategic changes brought on by the complementarity of massively available data and big analytics Data sets are already, and shall be more and more, very high-value-added assets. Their conservation and archiving shall be made easier by lower storage and storage management costs, together with affordable distributed access possibilities offered by Cloud Computing. Making such data assets available (not necessarily for free) is critical for the development of robust and new predictive models and in the end for the development of new business models. Tools such as cloud infrastructures, or national Big Data platforms dramatically decrease storage prices, data management costs and ease data sharing. But this is not enough to ensure data sharing. Data owners need to be able to either trust hosting infrastructures or have adequate tools to protect their data assets even in untrusted environments. 14

15 Therefore data privacy and security is now the main hurdle which prevents data owners from joining Big Data innovation environments. A similar statement can be made for citizens for whom privacy issues are taken more and more seriously 9. Data anonymization is therefore a key technical challenge to be addressed in this cppp. The main economic differentiators in the coming years for companies will lie in the ability to perform increasingly complex analytics at the timing dictated by their business needs. While processing frameworks currently exist for both large scale batch and complex processing of data at rest and high speed stream and simple processing of data in motion, it is only when reference architectures are designed to combine both approaches and disseminated across the different economic sectors, that highly complex analytical solutions with acceptable response time can rapidly emerge and be easily deployed in these sectors. Among analytic technologies, a key issue in the coming years will be predictive modeling and deep learning techniques. Such techniques, if we are able to apply them at scale and for data in motion can play a crucial to extract knowledge out of the data and to develop decision support applications. This requires new approaches for scalable data collection and integration, data processing, analysis, and compression, enhanced technologies for processing and sharing data and information, as well as novel languages for the declarative specification and automatic optimization, parallelization and hardware adaptation of advanced data analysis programmes in particular when considering advancements and new developments in processing, memory/storage, and networking technology. The recent moves of large American companies in the field, with the recruitment of hundreds of academics & engineers, a large part of them working in Europe, clearly illustrates that Big Analytics is, first, still a field open to competition, second, a field in which Europe has strong competitive advantages and third, a promising field for business development, at least is it considered as such by IT companies. A second and highly challenging issue will be graph mining techniques, applied on extremely large graphs. Because graph data are so widespread and affect almost all sectors, like transport, retail, media, health, energy, logistics, it should definitely be considered as a high priority challenge. To maximize the impact of Big Analytics, it is important not only to make data assets available, but also to ensure that various data assets can be easily integrated in the same processing framework. Because value will emerge out of the combination of several data assets, data integration, at a large scale, must be considered as a serious issue. A large part of the future Big Analytics solutions will be driven towards the enhancement of decision support capabilities. Because the most advanced predictive and deep analysis algorithms will always need to be put in humanunderstandable context to provide effective and trustworthy decision support, usability and the optimization of user experience remains a key issue. In the context of Big Data applications, in which the amount, velocity and heterogeneity of the processed data are very high the need is critical for highly interactive visualization techniques able to provide near-real time updates of the situation. However, because such a challenge can be first tackled, to a certain extent, 9 Recent court actions undertaken by citizens against IT companies are a good illustration of this trend. 15

16 through the integration of ergonomics best practices from the early stages of each of the projects in the cppp, this challenge can maybe be considered of a lower priority than what the simple needs analysis may suggest. Needs analysis In order to assess the needs of Big Data solution end users, sectorial workshops have been performed in various fields: geo-spatial/environment, energy, media, mobility, manufacturing, retail, health, public sector. From the analysis of the collected needs it is clear that to address the vertical application markets technical needs, a set of cross-sectors technologies are needed. The key technical topics most often mentioned in these sectorial workshops are the following: Data protection and privacy technologies: to make data owners comfortable about sharing data in an experimental environment; Advanced data mining: predictive analytics, graph mining, semantic analysis Low latency and real-time data processing Harmonization across different sources: standardized modeling, integration of heterogeneous data sources Advanced visualization, user experience and usability The impact of such transverse technologies goes well beyond the vertical sectors described as they require an "ecosystem" that will bring together stakeholders from the European Big Data community from both demand and suppliers sides, from legal, societal and technical sides. Besides, different technical needs and concerns exist according to the role of the stakeholder in the data ecosystem. Three main roles can be identified in the data ecosystem 10 : Role What do they do? How do they make business? Main technical concerns, needs Data provider Collect, pre-process, transform data into information and sell or distribute the information Make a margin on the resale of information Data curation and integration from heterogeneous sources. Data processor and service provider Buy information, perform deeper analysis to create value and provide services. ROI = revenues costs > 0 Leverages scale effects across multiple clients, service fees Need for low latency and data analytics at low cost, tools; flexibility to serve multiple clients, wide variety of data sources, sinks Service consumer Buy/use services Applies decisions and insights derived from analysis to Privacy and anonymisation. 10 Note that a given player of the ecosystem can in turn play several roles, consuming a service which allows him to provide another enriched service or data asset to another player. 16

17 optimization of own business Table 1. Roles and activities of ecosystem actors Data Data Provider Extract value Information Data Process or / Service Create value Services Service consume r Privacy Anonymisation Data curation (preprocessing, quality including veracity, integrity, lifecycle management) Data availability (cloud issues, low latency infrastructure & storage) Data Integration (Linking data assets) Meaningfulness of data Standardized modelling data formats & meta-data models Predictive analytics Data visualization & advanced search Low latency & scalability processing Data Integration (linking data) User experience, usability (data visualization; advanced search; data discovery) Decision support Figure 2. Various technical needs and concerns according to the role in the ecosystems The figure above depicts the technical concerns according to the three main roles identified in the data ecosystem. There are mainly two types of technical issues. Some existing challenges become more critical and difficult due to the increasing complexity and need for lower latency. However, new opportunities to solve problems that couldn t be solved previously also arise due to the availability of novel technology. For instance, crowd sourcing is generally associated with a high risk for data integrity while it could actually be used as a novel means to perform consistency checks on information from different sources, therefore leading to enhanced data quality, management and integrity solutions. There are challenges to cope with the volume, velocity, variety, and veracity aspects of data analytics and to integrate novel statistical and mathematical algorithms, as well as prediction techniques into services and applications. This requires new approaches for engineering of data management solutions, advanced technologies for visualisation, veracity checks, anonymisation and deep analysis. All those are considered as strategic priorities. 17

18 Topics such as data accessibility, low latency infrastructure and storage that are related to cloud computing issues will be handled in other complementary initiatives or will be covered by existing business activities. Overall technical goal of the cppp In the coming years the real complexity will consist in being able to deliver shorter and shorter response time over more and more complex systems and data sources. Organizations able to handle the increasing complexity and dynamicity of data structure and operations will thus gain a clear competitive advantage. Based on the needs analysis, the technical objectives of the cppp correspond to what is needed to manage, analyze and interpret very large amounts of heterogeneous, complex and dynamic data. Besides, a discriminating criterion for technological choices is clearly the latency to respond (data velocity) to requests. Response time is also a key differentiating economic criteria. In this perspective, therefore, the overall technical goal of the cppp will be to: Provide sufficient anonymisation guarantees, optimized user experience support and a sound data engineering framework allowing for deepened analysis capacities on renewed architectures. This requires addressing various technical research and innovation priorities: Privacy and anonymisation mechanisms Deep analysis to improve data understanding, deep learning, meaningfulness of data Optimized architectures for analytics of both data at rest and in motion, low latency and scalability processing, real-time analytics Advanced visualization and user experience Engineering the management of data. Leading to 5 technical priorities The following sections go in more details throughout the major technical challenges. Priority : Privacy and anonymisation mechanisms Title Research challenge Privacy and anonymisation mechanisms Data usage should be conform to the agreed policy. This is difficult to be assured. On the technical side, mechanisms are needed in order to provide the data owner means to control data access its usage and its lifecycle (citizens for instance should be able to 18

19 Rationale for selecting Potential outcome, innovation decide on the destruction of their personal data). In an ideal case, big data and data analytics are operated while protecting privacy and anonymity of data in order to gain maximum value. With the integration of multiple data sources, the chances to offer opportunities to crack and reverse anonymisation increase. Ensuring irreversibility of the anonymisation of Big Data assets is a key Big Data issue. Also, scalability of the solutions is a critical feature. Preserving anonymity often implies removing links between data assets. However this approach to preserve anonymity has to be reconciled with the needs for data quality, as removing links may negatively impact on data quality. If anonymisation per se is an important challenge it also implies other challenges, like for instance the need for analytics techniques to cope with anonymised data. Privacy and data anonymisation is one of the major concerns in the area of big data and data analytics involving all stakeholders in the value chain. Data privacy and security is indeed now the main hurdle which prevents data owners from joining Big Data innovation environments. A similar statement can be made for citizens who are more and more seriously taking into account privacy guarantees. Generalisation of Secure Remote Data Access Centre (CASD) techniques for data protection. This approach applied in the statistics domain has been successfully implemented in the French innovation space: Teralab. Methods for deletion of data and data minimization. Robust anonymisation algorithms Priority : Deep analytics Title Research challenges Rationale for selecting Deep analysis to improve data understanding Understanding data, be it numbers, text, or multimedia content, has always been one of the greatest challenges for IT. Entering into the era of Big Data this challenge has scaled to a degree that makes the development of new methods necessary. With respect to Big Analytics, key research issues are predictive modelling, deep learning techniques and graph mining techniques applied on extremely large graphs. As graph data are so widespread and affect almost all sectors, like transport, retail, media, health, energy, logistics, it should definitely be considered as high priority challenge. Another priority should be to improve the scalability of the processing speed for the aforementioned algorithms. The progress of deep analysis of big data is expected to have the most significant impact, as it will make big data much more usable, 19

20 Potential outcome, innovation and accessible to the wider public. Therefore it is expected to influence positively all parts of the value-chain and increase not only business opportunities but also societal benefits. Besides, deep Analytics is still a field open to competition, in which Europe has strong competitive advantages and which is promising for business development, at least it is considered as such by IT companies. Europe, being at the forefront of the big data revolution, will benefit by obtaining a largest share of the economic gains possible from the data-driven economy, and increased technological capabilities that will help shape future developments with respect to big data management. It was estimated that governments in Europe could save $149 billion by using Big Data analysis to improve operational efficiency. [McKinsey Global Institute (MGI): Big data: The next frontier for innovation, competition and productivity. 2011]. At a smaller scale, big analytics can provide additional value in every sector where it is applied, leading to more efficient and accurate processes. A more recent study, also by McKinsey Global Institute, has put an even stronger emphasis on this analytics issue, ranking it as the future main driver for the US economic growth, before shale oil and gas production [MGI: Game changers: Five opportunities for US growth and renewal, 2013] The main expected advanced analytics innovations are the following: Move beyond limited samples used so far in statistical analysis to samples covering the whole or the largest part of an event space. Improve the accuracy of statistical models by enabling fast nonlinear approximations in very large datasets. Discover rare events that are hard to identify since they have a small probability of occurrence, but have a great significance (such as physical disasters, a few costly claims in an insurance portfolio, rare diseases and treatments) Enable real-time analytics that are capable of analysing large amounts of data as they are produced and update the analysis results as the data changes. Deep learning, contextualization based on IA, machine learning, semantic analysis in near-real time, graph mining. Processing of unstructured data (multi-media, text): covering in particular multi-linguality Priority : Optimized architectures for analytics of both data at rest and in motion Title Research Optimized architectures for analytics of data at rest and in motion There have been advances for Big Data analytics to support new 20

21 challenge Rationale for selecting Potential outcome, innovation dimensions on Big Data volume (e.g. by NoSQL and Hadoop platforms). Separately stream processing has been enhanced to analytics on the fly to cover the velocity part of Big Data. This is especially important as business needs to know what is happening now. Being able to apply complex analytics techniques at scale and for data in motion plays a crucial to extract knowledge out of the data and to develop decision support applications. For instance predictive systems like recommendation engines must be able to provide real-time predictions while enriching historical databases to train always more complex and refined statistical models. Addressing this research challenge will benefit mainly the role of the Data Processor in the data ecosystem. While developing such solutions in an ad-hoc fashion is of course possible, only the design of generic, optimized architectural solutions and especially frameworks and toolboxes allowing for the best use of both data in motion and data at rest will leverage the dissemination of reference solutions which are ready and easy to deploy in any economic sector. When such solutions become available to service providers, in a straightforward manner, they will have the opportunity to focus on the development of business models. This research and innovation challenge is especially suitable to be addressed in a cppp as it requires the integration of up to now independently developed technologies together with very high demands on scalability for different aspects like volume, velocity and variety. Architectures, framework and tools for the integration of mostly existing components to new types of platforms, which address the orthogonal challenges in completely new ways by widening and generalizing known data processing capabilities for data at rest and data in motion. Priority : Advanced visualization and user experience Title Research challenge Advanced visualisation and user experience In the data visualisation domain the tools that are currently used to communicate information need to be improved because of a significant change in the relevant aspects of what should be communicated to the end users: focus on the adaptation to the needs of the end users (user adaptation and personalization but also advanced search capabilities) rather than on predefined visualization assets, focus on clusters rather than on individual categories, focus on relations and networks and therefore graphs rather than on simple comparisons, focus on positions and distances (geospatial data), 21

22 Rationale for selecting Potential outcome, innovation focus on qualitative analysis more than on quantitative ones, focus on dynamic 3D space, collaborative real-time solutions. Visualisation must consider the universe of data available from each private domain and current tools are not really adequate, because unknown and unpredictable data requires exploration capabilities over the visualisation. Access to information is at present essentially based on a user driven paradigm: the user knows what he needs and the only matter is to define the proper criteria. With the advent of big data, this user-driven paradigm does no longer prove to be the most correct and efficient answer, while a data driven paradigm emerges where information is extracted through data discovery techniques. We need an evolution of visual interfaces to become more intuitive and become able to cover the common and advanced aspects of Big Data analysis is required in order to foster effective exploitation of the information and knowledge that Big Data promises. Interactive, large graphs visualisation tools able to be updated in real-time Multi-modal and cross-lingual, scalable, real-time search engines User adaptable visualization tools with real-time performances, scalability, multi-language and multi tenant support that are able to combine any visualization asset in a plug-and play manner: maps, graph visualization, dashboards, parallel coordinates Priority : Data management engineering Title Research challenge Data management engineering All main actors involved in the data value chain, from producers to processors and consumers recognize the value and urgency of the definition of an effective and engineered approach to the management of data. That has been clearly highlighted as an output of the sector workshops as a strong request for more standardized formats and ability to meld together different datasets. However, given the history of standardizing data formats and semantics, it is unlikely that a common and widely used standard will emerge and be the definitive solution of the aforementioned problem. Collected data are rapidly increasing, but the efforts and management challenges do not evolve at the same pace as data growth and are affected by the lack of a broader and common agreement on the data management functions to include, and how to use and compose them in an easy-to-use way, according to a 22

23 Rationale for selecting Potential outcome, innovation clearly defined Data as a Service (DaaS) model. Tightly related research topics include: improving curation at scale with the evolution of human computation and crowdsourcing, general-purpose data curation pipelines, improved human-data interaction, standardized data curation models and vocabularies, and better integration between data curation tools. The industrialization process of data management is strategic because, although several different application sectors have already tried to develop vertical functions, such as specific data standards for years, and specific knowledge exists, the ability to clearly define how to access, transform, manage data is still missing. Data management mechanisms available and easily accessible as services. For instance, rather than focusing on the definition of a commonly agreed standard and the way to describe the meta-data, the PPP shall aim to give clear functions for managing the several different data (meta-data) standards and related management aspects. This includes novel current data management processing algorithms and data quality government approaches that support the specifics of Big Data. Data integration processes, including APIs for transforming one input into another, enabling also the possibility to plug and play different heterogeneous data sources; Harmonization of tools and techniques and the ability to easily re-use and bring to real life management use cases and services in cross-sectors, by diminishing the costs of developing new solutions; Tackling the whole data management lifecycle, from data curation (including pre-processing veracity, integrity, and quality of the data), to long term storage and data access; new models and tools to check integrity and veracity of data, through both machine-based and human-based (crowdsourcing) techniques. 23

24 Timeframe of the major expected outcomes The rationale for the timeframe is to work during the first years to remove silos and barriers of entry (such as security, integration of heterogeneous data sources, etc.). Then the final years correspond to topics which require longer term research. Year 1 Year 2 Year 3 Year 4 Year 5 Priority : privacy and anonymisat ion Generalisation of Secure Remote Data Access Centre techniques. Method for deletion of data and data minimization. Robust anonymisation algorithms Priority : deep analysis Improved statistical models by enabling fast non-linear approximation s in very large datasets Predictive modeling Graph mining techniques applied on extremely large graphs Real-time analytics Semantic analysis in near-real-time Algorithms for multimedia data mining Deep learning techniques Descriptive language for deep analytics. Contextualisatio n. Priority : architectur es for analytics of data at rest and in motion Optimized tools for the integration of existing components to new types of platforms with both data at rest and in motion. Synergies between massively parallel architectures (MPP) and batch processing/str eam processing architectures Priority : advance visualisatio n New data search solutions / paradigms Semantic driven data visualisation stronger links between visualization and analytics User adaptation Collaborative real-time, dynamic 3D solutions Priority : engineering data manageme nt APIs for improving the process of data transformation Collaborative Tools and techniques for Data Quality (including integrity and veracity check) Harmonized description format for metadata Methodology, models and tools for data lifecycle management Data management as a service Table 2: Timeframe of the major expected outcomes 24

25 8. Skills and Best Practices The amount, breadth and speed of data and thus the opportunities to turn this data into value has reached unprecedented levels over past years. Big Data, together with key enabling technology such as software engineering and cloud computing, has become a key driver for innovation and growth. Big Data software, services and applications generate value by fostering an innovation eco-system and by enabling completely new solutions that have not been possible before. The value lies in the applications base. The value lies in the application of advanced data-analysis on top of more general Big Data layers, also including semantic abstractions or network and physical objects virtualization 11. In order to leverage the potential of Big Data, a key challenge for Europe is to ensure the availability of highly and rightly skilled people who have an excellent grasp of the best practices and technologies for delivering Big Data Value technology and applications. This means that in addition to meeting the technical, innovation and business challenges as laid out in other Sections of this document, Europe needs to systematically address the need for educating tpeople that are furnished with the right skills and are able to leverage Big Data Value best practices. Education and training will play a pivotal role in creating and capitalizing on EU-based Big Data Value technologies and solutions. At the current stage of the Big Data discipline, we see two kinds of sub-disciplines emerging leading to two distinct breeds of specialists: Data Scientists and Data Engineers (as will be elaborated below). In fact, this is very similar to what happened to the software discipline in the years since the seminal NATO conference on Software Engineering in In software engineering there are now two principle, complementary types of specialists: (1) computer scientists, who are concerned with theoretical foundations and basic technology for creating software; (2) software engineers, who are concerned with establishing principles, tools, methods and sound engineering principles to efficiently and effectively develop, maintain and evolve software. Drawing on this similarity with the software field, we thus see a distinction between the following two kinds of Big Data specialists emerging 12 : Data Scientists: Successful Data Scientists will require solid knowledge in statistical foundations and advanced data analysis methods combined with a thorough understanding of scalable data management, with the associated technical and implementation aspects. They will be the specialists that can deliver novel algorithms and approaches for the Big Data Value stack in general, such as advanced learning algorithms, predictive analytics mechanisms, etc. For this, Europe needs new educational programmes in data science as well as ideally a network between scientists (academia) and industry that foster the exchange of ideas and challenges. Hence, innovation spaces could be used to a certain extent to build such networks. 11 NESSI White Paper on Big Data (2012). 12 Please note that, as always in novel fields, there are many different, even contrary definitions out there; e.g., some further consider a data analysts being a specific additional type of specialist (in our case we subsumed the competencies in our definition of data engineer); some flip the definitions of data engineer and scientists altogether. 25

26 Data Engineers: Those are the specialists that develop and exploit techniques, processes, tools and methods for developing applications that actually turn data into value. In addition to technical expertise, Data Engineers need to understand the domain and the business of the organizations. This means they need to bring in domain knowledge and are thus working at the intersection of technology, application domains and business. In a sense they thereby constitute the link between technology experts and the business analysts. Data Engineers will foster turning the development of Big Data applications from an art into a disciplined engineering approach. Data Engineers thereby allow the structured and planned development and delivery of customer-specific Big Data solutions, starting from a clear understanding of the domain, as well as customer and user needs and requirements. Data Engineers are probably the most important specialists in the future, emerging Big Data discipline, as Data Engineers foster delivering true value out of Big Data technology. Data Engineers will understand how to leverage the technology and foundations provisioned and developed by Data scientists. Ideally, this technology is available in packaged forms, i.e., in the form of frameworks, toolboxes and integrated models. This means that those frameworks, toolboxes, and integrated models bring individual solution components into a systems perspective (also see the discussion on technical issues in Section 7. In order to educate and train Data Engineers, novel courses and forms of training are required. That is, new university curricula, which are highly interdisciplinary and possibly very close to specific application domains, thus fostering acquiring domain knowledge. Such longer term university education should in addition be complemented by specialized trainings for the current workforce. Experience indicates that joint research & innovation projects serve as a good way to foster knowledge exchange between academia and industry, thus delivering experience about cutting-edge technology 13. Innovation spaces will play a major role for building competence for Data Engineers through the provisioning of data (private and open public) in a secure environment using open or proprietary platforms and new technology for analytics or applications. Innovation spaces thereby foster learning and best practice acquisition. In a sense, those innovation spaces may be considered labs where specialists can also play with and try out novel approaches towards Big Data Value generation, thereby gathering competence and best practices 9. Ecosystems, Business Models Big Data will have an impact in many sectors and domains including public sector, industry and individual citizens, and will do so along the primary dimensions of: efficiency, personalization, and new products and services: Big Data and data-driven decision making will help European companies and public organizations improve their operations and processes, gain efficiency with respect to their global competitors, and provide better services. 13 E.g., FP7 Objective 1.2 projects S-Cube, SEQUOIA, DEPLOY, PERSIST, RESERVOIR, and ServiceWeb3.0 report on well-trained graduates as a key outcome oft he projects. [This will also be elaborated in the forthcoming NESSI Software Engineering Whitepaper] 26

27 The collection and analysis of data from smart phones and IoT based environments will enable improvements and the creation of new services and business models focused around personalization. This is an area where both existing and new companies with data-centric business models will enter the market. Data-driven applications and the analysis of Big Data will help companies not only to design better products, and to create new business models but also to create completely new products. This is also a place where new companies with innovative business models, e.g. on collaborative data value creation will enter the market. Figure 3 The Data Value Chain Europe needs strong players along the Big Data Value Chain 14. It spans the range from data generation and acquisition, processing and analysis, curation, usage, and service provisions (figure. 2). All parts of the chain must be strong in order to make the whole Big Data Value ecosystem, and consequently Europe s economy, vibrant and valuable. Companies can occupy different positions in this ecosystem: Established User Enterprises, e.g. large enterprises that want to improve their services and products using Big Data technology, data products and services. Data Generators and Providers that create, collect, aggregate, transform and model raw data from various public and non-public sources and offer it to customers. Technology Providers that provide tools & platforms that offer data management and analytics tools to extract knowledge from data, curate and visualize it. Collaborative networks where different players in the value chain collaborate to offer value services to their customers based on data value creation,. 14 Michael.E.Porter: Competitive Advantage (1998) 27

28 These players contribute in different ways to the value chain, and a working ecosystem will need all of them. A core object of the PPP is to support specific instantiations of the value chain as well as the set-up this ecosystem. The specific obstacles in transforming an initial business idea to a sustainable business in this field will be: Established user enterprises: There is a technology-to-business gap, and experts able to translate technological advances in data analytics into precise, quantitatively well-defined business opportunities (and vice versa) are scarce. This is especially true when it comes to combine new products and business models with radical new technology and concepts. As a consequence, business opportunities might remain unidentified, or sub-optimal technological approaches are taken. Thus the disruptive opportunities lying at the heart of the Big Data revolution are in danger of being missed. Data Generators and providers: They often face the problem of creating an initial critical mass of customers to bootstrap their business. A further difficulty is, in many cases, insufficient exposure to new technology. Specific challenges in the European market stem from multi-linguality and privacy regulations as well as often needing to make microbusiness from the small data buried within the Big Data. Technology Providers often lack the domain knowledge needed to enter a market segment. Collaborative networks of data service providers that collect data from different sources and create value on top of it have to deal with legal challenges related to usage conditions of data, legal and privacy constraints. Additionally, defining a sustainable business model for these new services based on new technology is often a challenge, because this is often highly unexplored terrain. The purpose of the PPP and the Innovation Spaces is to address these problems and to help the various participants of the ecosystem to maximize their success by becoming part of the ecosystem. Business Model Methodology. Some large enterprises have started to systematically screen their data assets and data generating processes to identify use cases and business opportunities. Sometimes this results in dozens or even hundreds of potentially value generating use cases. Stakeholder communities that comprise different business sectors are established to foster information exchange and best practice sharing. They use a mix of tools ranging from traditional consulting, consultation workshops and employee idea competitions to the utilization of social networks. In some cases, these activities are embedded in an explicit methodological framework. The Innovation Space concept will transfer this approach to a European, crossenterprise, cross-sector level approach comprising all stakeholders of the Big Data Ecosystem. The structured approach to be developed in the PPP will allow all stakeholders to map their respective needs and offerings into a common landscape, so that business opportunities and technology options can be identified and mapped more quickly. Based on this, ecosystems will be established around viable business models. Secure Environment for testing Business Models. The PPP will be a meeting place for the partners in the ecosystem. At the core is a secure environment with access to Big Data for the partners. This helps the data providers get in contact with 28

29 potential customers. The customers themselves can explore their own business models using the data and services defined on top of the data, and the technology providers can generate showcases to demonstrate the value of their tools. The environment will offer a test-field, where new business approaches meet new technology and where entrepreneurs, decision makers from established enterprises and researcher will jointly bridge the gap between advanced technology and business models. New players will enter the value chain and establish new concepts and business models. Existing players will discover new possibilities to develop their incumbent business. Sustainability and viability requires that the commercial values are precisely defined and agreed between the parts of the value chain. Data service clearing house. Privacy aspects and compliance with legal regulations are one of the biggest challenges for a successful data marketplace. The PPP will address these issues in a systematic way and offer support to companies, especially SME s, in terms of best (legal) practice and advice. 10. Input to Regulation and Standardisation A favourable Regulatory Environment will be paramount to foster take up of Big Data Value technology and solutions. It is perceivable that the Big Data developments will accelerate in many of its aspects including collection, storage, analysis and use of data with increased impact on society and will require an in depth discussion among stakeholders to find the most effective solutions. Europe has a comprehensive set on regulation that is not yet fully tuned towards the developments of Big Data. Yet discussions are under way to update regulations such as for data protection and access to public data. The General Data Protection Regulation proposal - currently under negotiation - aims to address important privacy aspects stemming out from latest technology developments (including social media and cloud computing), and provide for harmonization of regulations throughout the EU. Advancements in the area of access to public data will include amongst others the creation of a genuine right to re-use public information (libraries, museums and archives and making data available in machine-readable and open formats. Indeed Europe needs to further work on a globally competitive framework for data privacy, security, IP, ownership and liability, which would leave flexibility to business to develop innovation while preserving privacy and security of individual citizen and businesses. Europe has a great opportunity for global leadership with a legislation that both renews its commitment to privacy and embraces innovation. Against this background, the cppp will play a role in fostering the exchange between data generators and services providers, so to suggest solutions on how to balance the legitimate interests of people and the needs for innovative services and solutions. All stakeholders active in the partnership will be able to contribute, sharing their perspective on the regulatory challenges they currently face, and on how services can provide enough benefits to the individuals while ensuring proper protection of personal data. The cppp projects will serve as tools to identify challenges and possible solutions. In addition, specific fora of discussion should be set-up within the cppp with the aim of: providing recommendations to legislators on how to adapt the regulatory framework, to make it fit to embrace data-driven innovation. 29

30 giving information to data owners on the benefits of data sharing Issues to be addressed include: Identification of issues from data acquisition, ownership of raw data, processed data, augmented data, aggregated data; Liability of data analysis, to deal with potential damage caused by incorrect analysis Measurement the value of data, the succession of data rights and the legacy of stored data for new, merged or bankrupt companies; Delete mechanisms, policies for data processing and data usage. Identifying the barriers with regard to ownership of data and intellectual property rights incl. copyright and access rights Standardisation is essential to the creation of a Data Economy and the cppp will support establishing and augmenting both formal and de-facto standards. The cppp will achieve this by: Leveraging existing common standards as the basis for an open and successful Big Data market Integrating national efforts on an international (European) level as early as possible Ensuring availability of experts for all aspects of Big Data in the standardisation process Providing education and education material to promote developing standards Standards play a pivotal role on any market to provide customers with true choice by being able to choose comparable and compatible goods or services from multiple suppliers. In the Big Data ecosystem this applies to both the technology and to the data. Technology Standardisation: Most technology standards for Big Data processing technology are de facto standards that are not prescribed (but at best described after the fact) by a standards organisation. The most prominent example is, of course, Hadoop and Map/reduce. However, the lack of standards is a major barrier. The history of NoSQL is based on solving specific technologies challenges that lead to a range of different storage technologies. The large range of choices coupled with the lack of standards for querying the data makes it harder to exchange data stores as it may tie application specific code to a certain storage solution. The NoSQL databases are designed for scalability, often by sacrificing consistency. Compared to relational databases, they often use a low level, non-standardized query interface that makes it harder to integrate in existing applications that expect an SQL interface. The lack of standard interfaces also makes it harder to switch vendors. While it seems plausible to define standards for a certain type of NoSQL databases, creating one language for different NoSQL database types is a hard task with an unclear outcome. The PPP would take a pragmatic approach to standardisation and would look to influence the standardisation of technologies such as Complex event processing for real-time Big Data applications, languages to encode the extracted knowledge bases, computation infrastructure, data curation infrastructure, query interfaces, and data storage technologies. 30

31 Data Standardisation: Data variety of Big Data makes it very difficult to standardise. Nevertheless, there is a lot of potential for data standardisation in the areas of data exchange and data interoperability. Big Data is valuable for any organisation across many sectors. Exchange and use of data assets is essential for functioning ecosystems and the data economy. Enabling the seamless flow of data between participants (i.e. companies, institutions, and individuals) is a necessary cornerstone of the ecosystem. Enabling interoperability within the ecosystem the removing of technology barriers are discussed in the last section, but also overcome the conceptual barriers of the data syntactic (format) and semantic differences (interpretation of meaning) of exchanged information. To this end the cppp would undertake collaborative efforts to support, where possible and pragmatic, the definition of semantic standardized data representation ranging from domain (industry sector) specific solutions, like domain ontologies to general concepts such as Linked Open Data. If such standards for data descriptions and meta-data could be established, it would simplify and reduce the cost of data exchange. Insufficiently described data formats, which are a barrier for global & efficient data exchange and processing, are then eliminated. 11. Work Programmes and Instruments The Big Data Value cppp will be implemented with priorities and instruments that will follow a priority setting along candidates identified today and revised throughout the lifetime of the partnership. The multi stakeholder input and sector analysis that has been the basis for the definition of the strategic scope of the cppp is equally involved to operationalize the instruments and sectors to be addressed within the first work programme. The Stakeholder Platform will be the major instrument to update the programme and define priorities in subsequent cycles. From a number of workshops an initial set of supply and demand data assets emerged that provided for the selection of candidates for the priority setting for the initial work programme. The work programme will comprise a variety of instruments including small to large research and innovation -, innovation -, support actions - and light house projects to support the strategy to create impact in areas with highest demand and fastest turnaround time. The operational priority (candidates) follow the initial analysis of European sectors with economic relevance. Programme Structure The Big Data Value cppp will comprise a variety of instruments and projects that will need to collaborate together, exchange results between each other and to coordinate joint activities e.g. within innovation spaces. Coordination amongst the cppp projects will be essential to create a self-reinforcing set of activities, provide for the development of common project strategies and the creation of a joint branding and harmonised perception in the community and beyond. A Stakeholders Platform project will continue (in same or similar form) as set up by FP7 (BIG) and Horizon 2020 (Byte) to provide for guidance and governance of the programming cycle. It will start with the cppp implementation revise priority settings 31

32 and will run through until the end of the programme with a final evaluation. The Stakeholder Platform will be set up and maintained in the most neutral form for maximum credibility on community and at the stakeholder level e.g. Member States. The Innovation Spaces projects will follow the experience gained in first call ICT15 project about setting up and running Big Data Value incubators for providing participant to learn, build skills and most importantly provide for secure environments where public open data can be linked to private sensitive data. They should start immediately to be able to build up the communities and federations that are needed to share best practise and provide access to data not available on specific labs/incubators. Incentivisation for creating new innovation spaces or involving regional/structural funds will be part of the pan-european federation exercise. Lesson learnt by other European incubators, innovation labs such as FI PPP and FIRE will be incorporated. Projects for investigating and evaluating Business Models will be connected to the innovation spaces where suppliers and users will meet and ideally the main parts of a specific Big Data Value chain are present. Social impact studies will be crucial as they will discover what measures need to be in place to build trust and confidence in the data economy for end users, small and large enterprises and public sector equally. They should run simultaneous to other projects in environments such as the incubators to have access to applications and use of e.g. simulations. Picking up on results of previous projects and developments in the standardisation arena cppp projects should contribute either as integral part of their work or as support actions to achieve a maximum of interoperability, transportability and standardisation of data. Starting promptly will by most beneficial. The novel value structures and characteristics of data in Big Data ecosystems will require an adequate regulation and therefore swift actions form the regulatory bodies. The cppp project targeting to contribute to the development of favourable regulatory environment should as well start from the beginning. The R&I / I projects will follow the priorities derived from the sector and data asset analysis and will be adapted to the need in time and complexity. The Lighthouse projects will play the most prominent role within the Big Data Value cppp for reasons of creating awareness of the potential, demonstrating the viability of the new data economy and testing new business models and building up new ecosystems. They also will represent the single biggest projects due to the size and involved activities. The following graph depicts the possible structure the activities of a BigDataValue cppp could take. 32

33 Figure 4 The envisaged Project Structure of the cppp Programme Instruments The Big Data Value cppp will be implemented with instruments that will follow a priority setting along candidates identified today (see description of challenges pointed out in this SRIA). Instruments have been defined based on the needs of such Big Data Value challenges so that they can serve the purpose of on the one hand developing those technologies that are needed to extract value from data and on the other hand facilitate adoption and market uptake. A brief description of such instruments is provided here: Research and Innovation Actions (RIA) will be collaborative projects with a duration typically ranging between 2 and 4 years and will be characterized by the technical excellence. They should be data-driven and demand-driven in the sense that they should serve a concrete purpose, addressing clear end-users business cases and well identified problems. They should show evidence on data value, making clear the way those technologies are contributing to improve or change the way data are managed, stored, visualized, and most important, how data is exploited by the enduser in the defined scenarios. Projects should be accompanied by measurable KPIs and define their development cycles in a way that allows availability of results as early as possible in the project and iterative testing within real scenarios. Feedback from user communities should be then fed into the process to facilitate validation and finetuning the solutions. Innovation actions (IA): Challenges in Big Data Value range from short to long term priorities and the cppp has to guarantee that there are instruments that appropriately respond to the different challenges in terms of urgency and timing. IAs will be used to provide quick results that do not require new research, but adoption of existing solutions by European companies. IAs will capitalize on previous or existing work and will focus their effort on bringing solutions to the market. Their duration will need to be more flexible to jumpstart and address innovation topics swiftly. They will showcase the potential of Big Data in different environments from critical sectors for Europe. IAs should be focused and market-driven projects that involve the main players of the 33