ISO/IEC JTC 1 N 12414

Transcription

1 ISO/IEC JTC 1 N 11 ISO/IEC JTC 1 Information technology Secretariat: ANSI (United States) Document type: Title: Status: Text for NP ballot Proposal for a New Work Item on Information technology Big Data Definition and Vocabulary Please submit your vote via the online balloting system. Date of document: Source: Expected action: Korea VOTE Action due date: of secretary: Committee URL: lrajchel@ansi.org

2 NEW WORK ITEM PROPOSAL Closing date for voting Date of circulation Secretariat ANSI Reference number (to be given by the Secretariat) ISO/IEC JTC 1 NXXXXX Proposer: Korea A proposal for a new work item within the scope of an existing committee shall be submitted to the secretariat of that committee with a copy to the ITTF and, in the case of a subcommittee, a copy to the JTC 1 secretariat. Proposals not within the scope of JTC 1 shall be submitted to the ITTF. A new work item proposal may be made by a national body, the JTC 1 secretariat or JTC 1 subcommittee secretariat, another technical committee or subcommittee, a Category A organization in liaison with JTC 1, the technical management board or one of its advisory groups, or the Chief Executive Officer. The proposal will be circulated to the P-members of JTC 1 or JTC 1 subcommittee for voting, and to the O-members for information. IMPORTANT NOTE: Proposals without adequate justification risk rejection or referral back to the originator. Guidelines for proposing and justifying a new work item are given overleaf. Proposal (to be completed by the proposer) Title of the proposed deliverable (in the case of an amendment, revision or a new part of an existing publication, show the reference number and current title) Information technology Big Data Definition and Vocabulary Scope of the proposed deliverable The proposed Internal Standard will provide a definition of Big Data along with a set of terms in fields of Big Data. It is a terminology foundation for Big Data standardization works. Its scope will include: Big Data Definition and Taxonomies, Big Data Elements, Big Data Patterns. This project will not be addressing legal, regulatory, and cross-border trade discussions, but will help with, and provide a framework for, and deeper and better understanding of, issues involving such topics so that others in the international community can address them. Purpose and justification of the proposal (attach a separate page as annex, if necessary) The purpose of this publication is to create a standard which provides common terms and definitions for the field of Big Data. Justification: Big data is being defined and classified with different ways by industry, academia, and research institutes. This is a field in flux, and different people may have different conceptions of what terms mean. This impedes efforts to standardize Big Data because each specification provides its own definitions, often obscuring the terminology similarities and differences across specifications. This Internal Standard will clarify the underlying concepts of Big Data and Data Science to enhance communication among producers and consumers of these new technologies by helping us use the same term for the same concept. Is this a Management Systems Standard (MSS)? Yes No Envisaged publication type (indicate one of the following, if possible) International Standard Technical Specification Technical Report Proposed development track 1 ( months) ( months - default) (8 months) Target dates for availability First CD 1. October 01 Publication 1. October 018 Known patented items (see ISO/IEC Directives Part 1 for important guidance) Yes No If "Yes", provide full information in an annex. Are there any known accessibility requirements and/or dependencies? Yes No If yes, please specify in a separate annex. Are there any known requirements for cultural and linguistic adaptability? Yes No If yes, please specify in a separate annex. DRAFT Version 01-mm-dd JTC 1 New Work Item Proposal Page 1

3 Meeting Information Estimated number of meetings: 8 Frequency of meetings: Date and place of first meeting (if known): - per year A listing of relevant documents ISO 0:009, Terminology work Principles and methods ISO 80:00 Terminology work Harmonisation of concepts and terms ISO 101-1:011, Terminology entries in standards Part 1: General requirements and examples of presentation ISO 101-:011, Terminology entries in standards Part : Adoption of standardised terminological entries ISO/IEC Information Technology - Cloud Computing - Overview and Vocabulary A statement from the proposer as to how the proposed work may relate to or impact on existing work, especially existing ISO and IEC deliverables. The proposer should explain how the work differs from apparently similar work, or explain how duplication and conflict will be minimized. (attach a separate page as annex, if necessary) The proposed work will complement existing work in the area of terminology. It will build on content from several ISO standard and produce a profile that is aligned with Big Data requirements. The work will also consider related projects such as ISO/IEC Information Technology - Cloud Computing - Overview and Vocabulary. A simple and concise statement identifying and describing relevant affected stakeholder categories (including small and medium sized enterprises) and how they will each benefit from or be impacted by the proposed deliverable(s). The proposed deliverables will benefit the international Big Data community and vendor by facilitating the consistent development and providing common terms & definitions for the field of Big Data. Liaisons (A listing of relevant external international organizations or internal parties (other ISO and/or IEC committees) to be engaged as liaisons in the development of the deliverable(s).) TeleManagement Forum, OASIS A listing of relevant countries which are not already P-members of the committee. Preparatory work (at a minimum an outline should be included with the proposal; ensure that all copyright issues are identified) A draft is attached An existing document is attached An outline is attached. to serve as an initial basis It is possible to supply a draft by The proposer or the proposer's organization is prepared to undertake the preparatory work required Yes No Proposed Project Editor (include contact information) Kangchan Lee chan@etri.re.kr for KATS (South Korea) Name of the Proposer (include contact information) Suwook Ha, Kangchan Lee (ETRI) sw.ha@etri.re.kr, chan@etri.re.kr Supplementary information relating to the proposal (Comments of the JTC 1 or SC Secretariat ) This proposal relates to a new ISO/IEC document This proposal relates to the amendment of an existing ISO/IEC document This proposal relates to the revision of an existing ISO/IEC document This proposal relates to a multi-part standard consisting of 10 parts This proposal relates to the adoption as an active project of an item currently registered as a Preliminary Work Item This proposal relates to the re-establishment of a cancelled project as an active project This proposal requires the service of a maintenance agency. If yes, has a potential candidate been identified? Please identify This proposal requires the service of a registration authority. If yes, has a potential candidate been identified? Please identify This proposal is submitted with a CD for simultaneous NP and CD balloting Other: Voting information The ballot associated with this proposal comprises a vote on (check only one): Adoption of the proposal as a new project (Stage 10.99) Adoption of the proposal as a new project and the associated draft as a working draft (WD) (Stage 0.0) Adoption of the proposal as a new project and the associated draft as a committee draft (CD) (Stage 0.0) Other: It is proposed to assign this new item to: JTC 1/WG9 a new JTC 1 subcommittee Annex(es) are included with this proposal (give details) None. DRAFT Version 01-mm-dd JTC 1 New Work Item Proposal Page

4 Use this form to propose: a) a new ISO/IEC document (including a new part to an existing ISO/IEC document), or the amendment/revision of an existing ISO/IEC document; b) the establishment as an active project of a preliminary work item, or the re-establishment of a cancelled project; c) the change in the type of an existing publication, e.g. conversion of a Technical Specification into an International Standard. This form is not intended for use to propose an action following a systematic review - use ISO Form 1 for that purpose. Proposals for correction (i.e. proposals for a Technical Corrigendum) should be submitted in writing directly to the secretariat concerned. DRAFT Version 01-mm-dd JTC 1 New Work Item Proposal Page

5 Guidelines on the completion of a proposal for a new work item (see also the ISO/IEC Directives Part 1 and the associated ISO/IEC JTC 1 Supplement) a) Title of the proposed deliverable: Indicate the subject of the proposed new work item. b) Scope of the proposed deliverable: Give a clear indication of the coverage of the proposed new work item. Indicate, for example, if this is a proposal for a new publication, or a proposed change (amendment/revision). It is often helpful to indicate what is not covered (exclusions). c) Envisaged publication type: Details of the types of ISO/IEC deliverable available are given in the ISO/IEC Directives, Part 1 and/or the associated JTC 1 Supplement. d) Purpose and justification of the proposal: Give details based on a critical study of the following elements wherever practicable. Wherever possible reference should be made to information contained in the related Business Plan. 1) The specific aims and reason for the standardization activity, with particular emphasis on the aspects of standardization to be covered, the problems it is expected to solve or the difficulties it is intended to overcome. ) The main interests that might benefit from or be affected by the activity, such as industry, consumers, trade, governments, distributors. ) Feasibility of the activity: Are there factors that could hinder the successful establishment or global application of the standard? ) Timeliness of the standard to be produced: Is the technology reasonably stabilized? If not, how much time is likely to be available before advances in technology may render the proposed standard outdated? Is the proposed standard required as a basis for the future development of the technology in question, or for adoption in a future regulatory system? ) Urgency of the activity, considering the needs of other fields or organizations. Indicate target date and, when a series of standards is proposed, suggest priorities. ) The benefits to be gained by the implementation of the proposed standard; alternatively, the loss or disadvantage(s) if no standard is established within a reasonable time. Data such as product volume or value of trade should be included and quantified. ) If the standardization activity is, or is likely to be, the subject of regulations or to require the harmonization of existing regulations, this should be indicated. 8) If a series of new work items is proposed having a common purpose and justification, a common proposal may be drafted including all elements to be clarified and enumerating the titles and scopes of each individual item. e) Relevant documents: List any known relevant documents (such as standards and regulations), regardless of their source. NOTE: The following criteria f) and g) do not mandate any feature for adaptability to culture, language, human functioning or context of use. The following criteria require that if any features are provided for adapting to culture, language, human functioning or context of use by the new Work Item proposal, then the proposer is required to identify these features. f) Accessibility: Indicate here whether the proposed standard takes into account diverse human functioning and diverse contexts of use. If so, indicate how it is addressed in your project plan. Indicate how the guidelines of ISO/IEC Guide 1 (Guidelines for standards developers to address the needs of older persons and persons with disabilities), ISO/IEC TR (Information technology -- Accessibility considerations for people with disabilities -- Part 1: User needs summary), and ISO TR 11 (Ergonomics data and guidelines for the application of ISO/IEC Guide 1 to products and services to address the needs of older persons and persons with disabilities) have been implemented in the proposal, or why they are not deemed to be relevant. g) Cultural and linguistic adaptability: Indicate here if cultural and natural language adaptability is applicable to your project. If so, indicate how it is addressed in your project plan. Typical examples of requirements include: 1) for text or speech, the user shall be able to choose the natural language of input and output sentences and the language captured shall be identified; ) for character coding, the code shall be clearly identified for correct input and rendering; ) for sorted lists, linguistic user order expectations shall be respected (see ISO/IEC 11 International string ordering and comparison); ) cultural variations in the way concepts are perceived in different countries shall be respected; and ) input methods used in a given country shall also be supported. For a list of what is required in most IT products, see ISO/IEC TR 19 (Guidelines, methodology, and reference criteria for cultural and linguistic adaptability in information technology products) and ISO/IEC TR 1101 (Framework for internationalization). h) Liaisons: List the relevant external international organizations or internal parties (other ISO and/or IEC committees) to be engaged as liaisons in the development of the deliverable(s). i) Preparatory Work: When the proposer considers that an existing well-established document may be acceptable as a standard (with or without amendment), indicate this with appropriate justification and attach a copy to the proposal. In this case, provide the document publication date, implementation history and national/global adoption experience. DRAFT Version 01-mm-dd JTC 1 New Work Item Proposal Page

6 Error! Reference source not found. Information Technology Big Data Definition and Vocabulary Document type: Error! Reference source not found. Document subtype: Error! Reference source not found. Document stage: Error! Reference source not found. Document language: Error! Reference source not found. C:\Users\lrajchel\Desktop\JTC 1 to do\korea NPs\KNB_NWIP_Definition_and_Vocabulary_outline.doc Error! Reference source not found.

7 Contents Page 1 Scope...1 Normative references...1 Terms and definitions Terms defined in this International Standard...1. Terms defined elsewhere... Symbols and abbreviated terms... Overview of Big Data....1 General description of Big Data.... Definition of Big Data.... Key characteristics of Big Data Volume..... Variety..... Velocity..... Variability..... Veracity... Actors and Roles for Big Data Data Provider...8. Data Consumer...8. Big Data Application Provider...8. Big Data Framework Provider...8 Big Data Elements Data Elements...8. Dataset at Rest...8. Dataset in Motion...9. Analytics within Data Science...9. Big Data Metrics...9. Big Data Security and Protection Big Data Patterns Data Process Data Process Ordering Changes Transactions Data Analytics Time Window Storage Medium Changes...9 Annex A (Informative) Examples and Use case of Big Data Bibliography ii Error! Reference source not found.

8 Information Technology Big Data Definition and Vocabulary 1 Scope This Internal Standard will provide a definition of Big Data along with a set of terms in fields of Big Data. It is a terminology foundation for Big Data standardization works. Its scope will include: Big Data Definition and Taxonomies, Big Data Elements, Big Data Patterns Note - This project will not be addressing legal, regulatory, and cross-border trade discussions, but will help with, and provide a framework for, and deeper and better understanding of, issues involving such topics so that others in the international community can address them Normative references The following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO 0:009, Terminology work Principles and methods ISO 80:00, Terminology work Harmonisation of concepts and terms ISO 101-1:011, Terminology entries in standards Part 1: General requirements and examples of presentation ISO 101-:011, Terminology entries in standards Part : Adoption of standardised terminological entries ISO/IEC 188, Information Technology - Cloud Computing Overview and Vocabulary Terms and definitions.1 Terms defined in this International Standard.1.1. Big Data Analytics: analytical functions to support the integration of results derived in parallel across distributed pieces of one or more data sources. This is a rapidly evolving field both in terms of functionality and the underlying programming model. Error! Reference source not found.error! Reference source not found. 1

9 Big Data Engineering: the storage and data manipulation technologies that leverage a collection of horizontally coupled resources to achieve a nearly linear scalability in performance..1.. Big Data Models: logical data models (relational and non-relational) and processing/computation models (batch, streaming, and transactional) for the storage and manipulation of data across horizontally scaled resources..1.. Big Data Paradigm: the distribution of data systems across horizontally-coupled independent resources to achieve the scalability needed for the efficient processing of extensive datasets..1.. NoSQL: datastores and interfaces that are not tied to strict relational approaches. Alternately called no SQL or not only SQL..1.. Non-Relational Models: logical data models such as document, graph, key value and others that are used to provide more efficient storage and access to non-tabular datasets..1.. Schema-on-read: big data is often stored in a raw form based on its production, with the schema, needed for organizing (and often cleansing) the data, is discovered and transformed as the data is queried Terms defined elsewhere..1. cloud computing: paradigm for enabling network access to a scalable and elastic pool of shareable physical or virtual resources with self-service provisioning and administration on-demand [Recommendation ITU-T Y.00 ISO/IEC 188, 01]... framework: structure expressed in diagrams, text, and formal rules which relates the components of a conceptual entity to each other [ISO 118-1, 01]... Internet of Things: an integrated environment, inter-connecting anything, anywhere at anytime [ISO/IEC JTC 1 SWG Report, 01]... lifecycle: evolution of a system, product, service, project or other human-made entity from conception through retirement [ISO/IEC TR , 011] Error! Reference source not found.

10 ownership: legal right of possession, including the right of disposition, and sharing in all the risks and profits commensurate with the degree of ownership interest or shareholding, as demonstrated by an examination of the substance, rather than the form, of ownership arrangements [ISO 108-, 011]... relational model : a data model whose structure is based on a set of relations [ISO/IEC 8-1, 1999]... role: a set of activities that serves a common purpose [Recommendation ITU-T Y.0 ISO/IEC 189, 01]..8. security: all aspects related to defining, achieving, and maintaining confidentiality, integrity, availability, non-repudiation, accountability, authenticity, and reliability of a system [ISO/IEC 188, 008]..9. sensor: device that observes and measures a physical property of a natural phenomenon or manmade process and converts that measurement into a signal [ISO/IEC 918-, 01]..10. streaming data: data passing across an interface from a source that is operating continuously [ISO/IEC 198-, 011] 8 9 Symbols and abbreviated terms For the purposes of this International Standard, the following abbreviations apply: 0 1 MPP NoSQL RA RFID SQL Massively Parallel Processing Not Only Structured Query Language Reference Architecture Radio-Frequency Identification SQL Query Language 8 Overview of Big Data In recent years, the term Big Data has emerged to describe a new paradigm for data applications. New technologies tend to emerge with a lot of hype, but it can take some time to tell what is new and different. While Big Data has been defined in a myriad of ways, the heart of the Big Data paradigm is that is too big Error! Reference source not found.

11 (volume), arrives too fast (velocity), changes too fast (variability), contains too much noise (veracity), or is too diverse (variety) to be processed within a local computing structure using traditional approaches and techniques. The technologies being introduced to support this paradigm have a wide variety of interfaces making it difficult to construct tools and applications that integrate data from multiple Big Data sources. This report identifies potential areas for standardization within the Big Data technology space..1 General description of Big Data Big Data is used as a concept that refers to the inability of traditional data architectures to efficiently handle the new data sets. Characteristics that force a new architecture to achieve efficiencies are the dataset-at-rest characteristics volume, and variety of data from multiple domains or types; and from the data-in-motion characteristics of velocity, or rate of flow, and variability (principally referring to a change in velocity). Each of these characteristics results in different architectures or different data lifecycle process orderings to achieve needed efficiencies. A number of other terms (often starting with the letter V ) are also used, but a number of these refer to the analytics and; not big data architectures. The new big data paradigm occurs when the scale of the data at rest or in motion forces the management of the data to be a significant driver in the design of the system architecture. Fundamentally the big data paradigm represents a shift in data system architectures from monolithic systems with vertical scaling (faster processors or disks) into a horizontally scaled system that integrates a loosely coupled set of resources. This shift occurred 0-some years ago in the simulation community when the scientific simulations began using massively parallel processing (MPP) systems. In different combinations of splitting the code and data across independent processors, computational scientists were able to greatly extend their simulation capabilities. This of course introduced a number of complications in such areas as message passing, data movement, and latency in the consistency across resources, load balancing, and system inefficiencies while waiting on other resources to complete their tasks. In the same way, the big data paradigm represents this same shift, again using different mechanisms to distribute code and data across loosely-coupled resources in order to provide the scaling in data handling that is needed to match the scaling in the data. The purpose of storing and retrieving large amounts of data is to perform analysis that produces additional knowledge about the data. In the past, the analysis was generally accomplished on a random sample of the data. Big Data Paradigm consists of the distribution of data systems across horizontally-coupled independent resources to achieve the scalability needed for the efficient processing of extensive datasets. With the new Big Data Paradigm, analytical functions can be executed against the entire data set or even in real-time on a continuous stream of data. Analysis may even integrate multiple data sources from different organizations. For example, consider the question What is the correlation between insect borne diseases, temperature, precipitation, and changes in foliage. To answer this question an analysis would need to integrate data about incidence and location of diseases, weather data, and aerial photography. While we certainly expect a continued evolution in the methods to achieve efficient scalability across resources, this paradigm shift (in analogy to the prior shift in the simulation community) is a one-time occurrence; at least until a new paradigm shift occurs beyond this crowdsourcing of processing or data system across multiple horizontally-coupled resources. Big Data Engineering is the storage and data manipulation technologies that leverage a collection of horizontally coupled resources to achieve a nearly linear scalability in performance. New engineering techniques in the data layer have been driven by the growing prominence of data types that cannot be handled efficiently in a traditional relational model. The need for scalable access in structured and unstructured data has led to software built on name-value/key-value pairs or columnar (big table), documentoriented, and graph (including triple-store) paradigms. Non-Relational Models refers to logical data models such as document, graph, key value and others that are used to provide more efficient storage and access to non-tabular datasets. Error! Reference source not found.

12 NoSQL (alternately called no SQL or not only SQL ) refers to datastores and interfaces that are not tied to strict relational approaches. Big Data Models refers to logical data models (relational and non-relational) and processing/computation models (batch, streaming, and transactional) for the storage and manipulation of data across horizontally scaled resources. Schema-on-read big data is often stored in a raw form based on its production, with the schema, needed for organizing (and often cleansing) the data, is discovered and transformed as the data is queried. This is critical since in order for many analytics to run efficiently the data must be structured to support the specific algorithms or processing frameworks involved. Big Data Analytics is rapidly evolving both in terms of functionality and the underlying programming model. Such analytical functions support the integration of results derived in parallel across distributed pieces of one or more data sources. The Big Data paradigm has other implications from these technical innovations. The changes are not only in the logical data storage, but in the parallel distribution of data and code in the physical file system and direct queries against this storage. The shift in thinking causes changes in the traditional data lifecycle. One description of the end-to-end data lifecycle categorizes the steps as collection, preparation, analysis and action. Different big data use cases can be characterized in terms of the dataset characteristics at-rest or in-motion, and in terms of the time window for the end-to-end data lifecycle. Dataset characteristics change the data lifecycle processes in different ways, for example in the point of a lifecycle at which the data is placed in persistent storage. In a traditional relational model, the data is stored after preparation (for example after the extract-transform-load and cleansing processes). In a high velocity use case, the data is prepared and analyzed for alerting, and only then is the data (or aggregates of the data) given a persistent storage. In a volume use case the data is often stored in the raw state in which it was produced, prior to the application of the preparation processes to cleanse and organize the data. The consequence of persistence of data in its raw state is that a schema or model for the data is only applied when the data is retrieved, known as schema on read. A third consequence of big data engineering is often referred to as moving the processing to the data, not the data to the processing. The implication is that the data is too extensive to be queried and transmitted into another resource for analysis, so the analysis program is instead distributed to the data-holding resources; with only the results being aggregated on a different resource. Since I/O bandwidth is frequently the limited resource in moving data, another approach would be to embed query/filter programs within the physical storage medium. At its heart, Big Data refers to the extension of data repositories and processing across horizontally-scaled resources, much in the same way the compute-intensive simulation community embraced massively parallel processing two decades ago. In the past, classic parallel computing applications utilized a rich set of communications and synchronization constructs and created diverse communications topologies. In contrast, today, with data sets growing into the Petabyte and Exabyte scales, distributed processing frameworks offering patterns such as map-reduce, offer a reliable high-level, commercially viable compute model based on commodity computing resources, dynamic resource scheduling, and synchronization techniques.. Definition of Big Data The term Big Data is used in a variety of contexts with a variety of characteristics. To understand where standards will help support the big data paradigm, we have to reach some level of consensus on what the term really means. This report uses the following working definition of Big Data : Big Data is a data set(s) with characteristics (e.g. volume, velocity, variety, variability, veracity, etc.) that for a particular problem domain at a given point in time cannot be efficiently processed using current/existing/established/traditional technologies and techniques in order to extract value. Error! Reference source not found.

13 The above definition distinguishes Big Data from business intelligence and traditional transactional processing while alluding to a broad spectrum of applications that includes them. The ultimate goal of processing Big Data is to derive differentiated value that can be trusted (because the underlying data can be trusted). This is done through the application of advanced analytics against the complete corpus of data regardless of scale. Parsing this goal helps frame the value discussion for Big-Data use cases. Any scale of operations and data: Utilizing the entire corpus of relevant information, rather than just samples or subsets. It's also about unifying all decision-support time-horizons (past, present, and future) through statistically derived insights into deep data sets in all those dimensions. Trustworthy data: Deriving valid insights either from a single-version-of-truth consolidation and cleansing of deep data, or from statistical models that sift haystacks of "dirty" data to find the needles of valid insight. Advanced analytics: Faster insights through a variety of analytic and mining techniques from data patterns, such as long tail analyses, micro-segmentations, and others, that are not feasible if you're constrained to smaller volumes, slower velocities, narrower varieties, and undetermined veracities. A difficult question is what makes Big Data big, or how large does a dataset have to be in order to be called big data? The answer is an unsatisfying it depends. Part of this issue is that Big is a relative term and with the growing density of computational and storage capabilities (e.g. more power in smaller more efficient form factors) what is considered big today will likely not be considered big tomorrow. Data is considered big if the use of the new scalable architectures provides improved business efficiency over other traditional architectures, in other words the functionality cannot be achieved in something like a traditional relational database platform. Big data essentially focuses on the self-referencing viewpoint that data is big because it requires scalable systems to handle it, and architectures with better scaling have come about because of the need to handle big data Key characteristics of Big Data..1 Volume Traditionally, the data volume requirements for analytic and transactional applications were in sub-terabyte territory. However, over the past decade, more organizations in diverse industries have identified requirements for analytic data volumes in the terabytes, petabytes, and beyond. Estimates produced by longitudinal studies started in 00 [8] show that the amounts of data in the world is doubling every two years. Should this trend continue, by 00, there will be 0 times the amount of data as there had been in 011. Other estimates indicate that 90% of all data ever created, was created in the past years. The sheer volume of the data is colossal - the era of a trillion sensors is upon us. This volume presents the most immediate challenge to conventional information technology structures. It has stimulated new ways for scalable storage across a collection of horizontally coupled resources, and a distributed approach to querying. Briefly, the traditional relational model has been relaxed for the persistence of newly prominent data types. These logical non-relational data models, typically lumped together as NoSQL, can currently be classified as Big Table, Name-Value, Document and Graphical models. A discussion of these logical models was not part of the phase one activities that led to this document... Variety Traditionally, enterprise data implementations for analytics and transactions operated on a single structured, row-based, relational domain of data. However, increasingly, data applications are creating, consuming, Error! Reference source not found.

14 processing, and analyzing data in a wide range of relational and non-relational formats including structured, unstructured, semi-structured, documents and so forth from diverse application domains. Traditionally, a variety of data was handled through transforms or pre-analytics to extract features that would allow integration with other data through a relational model. Given the wider range of data formats, structures, timescales and semantics that are desirous to use in analytics, the integration of this data becomes more complex. This challenge arises as data to be integrated could be text from social networks, image data, or a raw feed directly from a sensor source. The Internet of Things is the term used to describe the ubiquity of connected sensors, from RFID tags for location, to smart phones, to home utility meters. The fusion of all of this streaming data will be a challenge for developing a total situational awareness. Big Data Engineering has spawned data storage models that are more efficient for unstructured data types than a relational model, causing a derivative issue for the mechanisms to integrate this data. It is possible that the data to be integrated for analytics may be of such volume that it cannot be moved in order to integrate, or it may be that some of the data is not under control of the organization creating the data system. In either case, the variety of big data forces a range of new big data engineering in order to efficiently and automatically integrate data that is stored across multiple repositories and in multiple formats... Velocity The Velocity is the speed/rate at which the data is created, stored, analyzed and visualized. Traditionally, most enterprises separated their transaction processing and analytics. Enterprise data analytics were concerned with batch data extraction, processing, replication, delivery, and other applications. But increasingly, organizations everywhere have begun to emphasize the need for real-time, streaming, continuous data discovery, extraction, processing, analysis, and access. In the big data era, data is created in real-time or near real-time. With the availability of Internet connected devices, wireless or wired, machines and devices can pass-on their data the moment it is created. Data Flow rates are increasing with enormous speeds and variability, creating new challenges to enable real or near real-time data usage. Traditionally this concept has been described as streaming data. As such there are aspects of this that are not new, as companies such as those in telecommunication have been sifting through high volume and velocity data for years. The new horizontal scaling approaches do however add new big data engineering options for efficiently handling this data... Variability Variability refers to changes in data rate, format/structure, semantics, and/or quality that impact the supported application, analytic, or problem. Specifically, variability is a change in one or more of the other Big Data characteristics. Impacts can include the need to refactor architectures, interfaces, processing/algorithms, integration/fusion, storage, applicability, or use of the data. The other characteristics directly affect the scope of the impact for a change in one dimension. For, example in a system that deals with petabytes or exabytes of data refactoring the data architecture and performing the necessary transformation to accommodate a change in structure from the source data may not even be feasible even with the horizontal scaling typically associated with big data architectures. In addition, the trend to integrate data from outside the organization to obtain more refined analytic results combined with the rapid evolution in technology means that enterprises must be able to adapt rapidly to data variations... Veracity Veracity refers to the trustworthiness, applicability, noise, bias, abnormality and other quality properties in the data. Veracity is a challenge in combination with other Big Data characteristics, but is essential to the value associated with or developed from the data for a specific problem/application. Assessment, understanding, exploiting, and controlling Veracity in Big Data cannot be addressed efficiently and sufficiently throughout the data lifecycle using current technologies and techniques. Error! Reference source not found.

15 Actors and Roles for Big Data.1 Data Provider A Data Provider makes data available to themselves or to others. The actor fulfilling this role can be part of the Big Data system, internal to the organization in another system, or external to the organization orchestrating the system. Once the data is within the local system, requests to retrieve the needed data will be made by the Big Data Application Provider and routed to the Big Data Framework Provider. Data Consumer The Data Consumer receives the value output of the Big Data system. In many respects, they are the recipients of the same functionality that the Data Provider brings to the Big Data Application Provider. After the system adds value to the original data sources, the Big Data Application Provider then offers that same functionality to the Data Consumer.. Big Data Application Provider The Big Data Application Provider executes the manipulations of the data lifecycle to meet the requirements. This is where the general capabilities within the Big Data Framework are combined to produce the specific data system.. Big Data Framework Provider The Big Data Framework Provider has general resources or services to be used by the Big Data Application Provider in the creation of the specific application. There are many new components here from which the Big Data Application Provider can choose in utilizing these resources and the network to build the specific system Big Data Elements The rate of growth in the amount of data generated and stored has been increasing exponentially. Data growth rates are considered to be more than Moore s Law would indicate if applied to data volumes - implying a doubling in volume every three months. This data explosion is creating opportunities for new ways of combining and using data to find value. One of the significant shifts is in the amount of unstructured data. Structured data has typically been the focus of most enterprise analytics, and has been handled through use of the relational data model. Micro-texts, relationship data, images and videos have seen such an explosion in quantity that the trend is to try to incorporate this data to generate value. The central benefit of Big Data analytics is the ability to process large amounts and types of information. Big Data To understand the components of Big Data, and to characterize the terms that are associated with Big Data and data science, the concepts will be addressed by category. 1.1 Data Elements. Dataset at Rest 8 Error! Reference source not found.

16 . Dataset in Motion. Analytics within Data Science Big Data Metrics. Big Data Security and Protection Big Data Patterns 8.1 Data Process Data Process Ordering Changes Transactions 1 8. Data Analytics Time Window 8. Storage Medium Changes Error! Reference source not found. 9

17 Annex A (Informative) Examples and Use case of Big Data 10 Error! Reference source not found.

18 Bibliography [01] Error! Reference source not found. 11