Understanding traffic flow
|
|
- Christian Reed
- 7 years ago
- Views:
Transcription
1 White Paper A Real-time Data Hub For Smarter City Applications Intelligent Transportation Innovation for Real-time Traffic Flow Analytics with Dynamic Congestion Management
2 2 Understanding traffic flow and reducing congestion is an on-going challenge for the transportation industry. Adoption of new technology is helping, for example, by providing more frequent and detailed data using cheaper GPS devices and networks of wireless sensors. With approximately 800 million vehicles on the world s roads today, estimated to increase to four billion vehicles by 2050, reducing congestion will require operational systems and web application architectures with the performance required to process a vast volume of sensor data, but also with the capability to automate responses to changing conditions in real-time. This paper discusses the emergence of new Big Data technologies, and how these platforms can be harnessed to deliver real-time and actionable operational intelligence from streaming sensor, GPS and smartphone data. SENSOR BIG DATA STREAM PROCESSING. A DRIVER FOR CHANGE Streaming analytics with real-time data integration can improve an organization s ability to transform raw sensor data into actionable information. The challenge is to deliver a scalable real-time architecture that integrates sensor data with existing systems, and to deliver real-time information that can be acted on with confidence. Continuous, real-time integration also enables duplicated systems, databases, and data warehouses to be rationalized, and with fewer systems, organizations can operate with lower costs yet operate a more effective, real-time IT platform. Detecting an event such as a traffic incident as it happens is important, but it is better to predict congestion in advance and to offer the correct information so that avoiding actions can be taken. This could include dynamic updates to speed signs and automatic adjustment to traffic light phasing. Today s traffic management systems have the capacity to monitor some aspects of the road networks in real-time, but are unable to scale to complete network coverage, or to deliver accurate avoidance information and real-time predictive analytics. This next step requires a step change in both approach and in the underlying software system technology.
3 3 REAL-TIME SMART SERVICES Smart Services go beyond the kinds of upkeep and upgrades a company may practice internally and towards its customers. To create them, intelligence is needed that is, awareness, connectivity and real-time analytics alongside the products and services offered. Most importantly, action needs to be taken according to what the production/buying cycle reveals about itself- Smart Services are the result of systems taking intelligent actions in real time. WHAT MAKES SMART SERVICES SMART? Smart Services are completely different from the offerings based on traditional stored data technology: Predictive, rather than reactive. The flurry of data coming from different sources and integrated in real time with historic data provides actual evidence that an equipment is about to fail, inventories are too low for the weekend or that traffic will delay a delivery Reliable for customers or users, Smart Services add the value of removing unpleasant surprises. Companies can now calculate in real time product performance and customer behaviours, and focus the targeting strategies with unprecedented accuracy Efficient. In a Smart Services environment, machines and devices do what they are very good at doing: producing and digesting billions of data points, talking to one another about the data, controlling one another based upon the state of the data all in a matter of milliseconds. Humans cannot do this, nor should they; this continuous stream of business information is the feed for today s stream processing technologies. Provided this wealth of information is exploited when it s produced and, more importantly, when it is needed, managers and decision makers can gain much more visibility into a business s assets, costs, and liabilities- and decide on the right triggers for automation.
4 4 Big Data Technologies The term Big Data is used to describe datasets whose volume, velocity, and variety are beyond the ability of traditional database and data management systems to capture, store, manage, and analyze The emergence of streaming Big Data technology is focused on the challenge of managing high volume, high velocity streams of data, transforming these into actionable information, and responding in a timely, predictable, and reliable manner. However, the emergence of Big Data technology is made possible by other factors such as Cloud computing and the availability of much cheaper server hardware and storage platforms. Big Data is not just about volume and velocity. In fact, the volume of data created each year exceeds the world s global storage capacity. Furthermore, the rate of increase in data creation is faster than the rate at which storage capacity is expanding. The transportation industry is at the forefront of the next generation of sensor network management and the exploitation of streaming Big Data analytics. This raises two questions. First, how can transportation industry exploit its Big Data? Secondly, if there is more data being created than can be processed by existing traditional database-oriented systems, how can a Big Data asset be exploited? BIG DATA Is defined by Volume, Velocity and Variety.
5 5 The data management technologies underpinning Big Data and streaming data processing are not new. As illustrated in Fig. 1, the initial model for data storage was the sequential data model, where data was stored as a sequence of data records with indexed access. The sequential model evolved into the hierarchical model for record databases, where complexity was managed by storing data in hierarchies, for example, IMS from IBM. Next came the major step forward in the form of the relational model, originated from IBM s System R project. This project also included the SQL language, a high level declarative model that has become the standard for data management. SQL is a specification for the problem to be solved that lets the underlying execution platform determine the more efficient way to compute the answers, including automatic optimization and management of distributed processing. So how has the history of data storage and management influenced Big Data technology? First, the original indexed file sequential model has returned in the form of name:value storage systems that underpin static Big Data storage platforms. Second, and more importantly, the SQL model is now being adopted as the de facto query language for Big Data management. Figure 1: The Evolution of Big Data Technology Big Data storage platforms are based on an open source software ecosystem called Hadoop, managed by the Apache Software Foundation. Hadoop was designed to overcome the two main limitations of traditional RDBMS technology for processing Internet data the ability to manage data with different formats and structure on the same platform and the ability to scale out over multiple servers for massively high performance. The core Hadoop parallel processing infrastructure was the enabler for new types of NoSQL databases ( Not Only SQL ) that allow data management to be distributed across many hundreds or even thousands of commodity server, all processing in parallel. However, as Big Data storage technology matures, SQL is now increasingly being added to the Big Data management portfolio for its querying power and ease of application building.
6 6 Quisque volutpat erat vel dolor. Maecenas leo. - Attribution Mauris The evolution of real-time data in motion technologies has paralleled that of static data management. The first effective data communication mechanism between systems was based on the concept of sockets, where each socket is essentially a logical address to which applications can send data and to which other applications can read the data. This evolved into messaging middleware, software applications responsible for the real-time and reliable delivery of data between multiple communication applications. Prior to the emergence of Big Data storage technology and Hadoop, stream processing technology was already emerging to address the requirement for processing real-time data. Stream processing also provides the capability to analyze and aggregate the data on-the-fly, as the data are created, and before the data are stored. As with Big Data storage platforms, SQL also emerged as the data management language for streaming data processing, that is, SQL as a continuous query language for streaming data. The Big Data industry has evolved to maximize the capability of existing technologies for both streaming and static data analysis. The volume, velocity and variety of sensor and other data in the transportation industry is sufficient to cause significant business and operational issues. Typical metrics given for Big Data problems include: Volume. The volume of data created in 2009 was 800 Exabytes, forecast to grow by 40% per year Velocity. Industry and sensor technology are at the forefront of Big Data velocity requirements. Large scale GPS applications are now exceeding 1 million events per second; however, telecommunications and IP-based services (Internet Protocol) monitoring applications require the capacity to process many tens of millions of events per second Variety. It is estimated that approximately 80% of all data in an organization is unstructured. This includes s and documents.
7 7 Big Data in the context of the transportation industry It is interesting to compare the scale of data processing capacities across different industries in order to better understand where Big Data technologies come into play. A Big Data system has become classified in terms of its ability to address the 3Vs of volume, velocity and variety, a definition originally attributed to Gartner Research. Data Volume. The total of GPS sensor data, fixed road sensor data, social media feeds, weather data, telematics and other location-based traveller information may exceed many terabytes of data per day, which must be processed both in real-time and stored for historical analysis. Data Velocity. Large-scale telematics applications for example can deliver Vehicle-to-Infrastructure (V2I) events at rates of many million of records per second. The average car generates between 5 and 250 gigabytes of sensor data an hour. Even if only a small percentage of the data is transmitted back through local access technologies, the core data processing platform must operate at the required speed for the complete network. To put this in perspective, Twitter messages during major sporting events peak at approximately 10,000 tweets per second. Data Variety. Unstructured and semi-structured data from sensors and from social media feeds such as Twitter are increasing and pose a significantly greater challenge for data processing and integration than conventional structured and sensor data. A BIG DATA STREAM PROCESSOR FOR REAL TIME TRAFFIC MANAGEMENT Real-time traffic management from streaming Big Data requires real-time monitoring, traffic analytics and automation, as well as the integration of traditional data sources: Per-segment historical speed/travel-time comparisons, for example, comparisons in real-time with the same period yesterday, last week or even last year Combine GPS sensor data streams in real-time with roadside camera and signal data, to achieve improved accuracy for congestion and travel time predictions Integration with roadside variable speed signs, providing dynamic adjustment of per-segment speed limits in order to respond in real-time to changing traffic conditions Driver and user access to the application, providing for example, real-time Travel Time through Smartphones and GPS devices Extend Travel Time application to provide end-to-end journey travel time across multiple transportation modes including heavy vehicles, rail, bus and ferry networks. However, the issues with operational traffic management platforms can be summarized as a lack of business integration across operational siloes, inability to manage the increasing volume and velocity of data, and a lack of real-time analysis and forecasting over the live sensor data feeds. The function of a streaming data management platform is to address these issues by offloading the real-time analysis and continuous integration while retaining the existing operational systems.
8 8 Stream processing is a paradigm for the continuous processing and transformation of real-time, dynamic data. Sensors are the most common source of streaming, real-time data, but any static data source such as a log files and databases can be instrumented using Change Data Capture (CDC) adapters to transform new updates into real-time streams. Change Data Capture refers to the ability to detect source data has changed and to capture the new data in real-time, as they are created. For example, capturing new records that have been added to a log file or new rows that have been written to a table in a database. The resulting streams of processed data and analytics are output to real-time dashboards in the operations center, traveller smartphone applications, transportation agency website applications, and pushed simultaneously to the existing database and data warehouse systems. The traditional approach of loading data into databases and data warehouses is referred to as Extract, Load and Transform (ETL). With a streaming data platform, data are aggregated on the fly (that is, analyzed as the data arrive before being persisted to a database or Big Data storage platform) and delivered in real-time using continuous ETL operations. This eliminates the high latency, slow update overhead that is commonplace with traditional batch-based ETL solutions. As illustrated in Figure 2, a streaming data management platform enables multiple applications to be deployed on a single core platform. Applications include data cleansing, monitoring and alerting, integration and visualization. Each application has access to any or all of the arriving data streams simultaneously. The platform assembles the streams that each application has requested, effectively sharing all the streams across any number of applications. Applications process the data streams using relational Views, where each View is generated by a continuously executing SQL query. Streaming SQL queries are active queries that execute over the live streaming data while the data are still in flight, without having to store the data first. Data and information is pushed out continuously to external systems as streams of results and processed data. Figure. 2: Data Stream Management Platform as a Real-time Data Hub for streaming applications
9 Real-time Platform for Traffic Flow Analysis with Dynamic Congestion Management The architectural paradigm for streaming data management has both similarities with and significant differences from traditional Relational Database Management Systems (RDBMS). Both RDBMS and streaming data management platforms are based on the industry standard SQL language; however, a traditional RDBMS must first store the data (data are persisted) before the data can be queried and analyzed. The primary differences between the two paradigms are described in Table 1. Query Duration Query Scope Query Federation Relational Streaming Platform Queries execute continuously Queries over arriving data Stream processing distributed inmemory over many server nodes RDBMS Queries complete and exit Ad-hoc queries over stored data Processing executed centrally over a single inmemory or disk-based repository Table 1: Streaming and traditional RDBMS SQL query comparison In summary, in a streaming data management platform, the arriving data streams are processed in-memory before the data are stored. Processing data entirely in main memory is significantly faster than processing stored data held on disk as it eliminates the performance bottleneck of data retrieval. The same RDBMS SQL queries can be deployed as streaming queries on a streaming data management platform. However, unlike an RDBMS query that always completes and returns a fixed data set, a streaming data SQL query is continuous and executes forever. Figure 3: Real- time traffic m anagement solutions from streaming GPS data with Twitter overlay THE ADOPTION OF BIG DATA AND REAL-TIME STREAM PROCESSING IS THE FUTURE TECHNOLOGY FOR TRANSPORTATION NETWORK MANAGEMENT. 9
10 10 Streaming data management is a complement to traditional RDBMSbased solutions. Both share the concept of a data model centered on processing relational rows, queries, and views. Both share common data manipulation and definition languages standardized as SQL. They are able to share a common security model and application programming interfaces, such as JDBC (Java DataBase Connectivity), and a common representation of metadata. A streaming data processor is based on predetermined queries executing continuously over arriving data, while an RDBMS is used for ad hoc queries over historical, stored data, processing each query until it terminates. Transactional processing is supported in both an RDBMS and a streaming data management platform. In an RDBMS, transactions mark the start and end of an update operation; in a relational streaming platform, the transactions delineate the arrival and delivery of data. The next generation of data processing architectures must utilize the strengths of different data management technologies. Examples are traditional RDBMS for data warehousing and Master Data Management (MDM), Big Data and NoSQL technologies for offline, batch-based pre-processing, and streaming data integration with inmemory analytics for the real-time, intelligent integration fabric across all operational systems. STREAMING BIG DATA IS THE COMPLEMENT TO TRADITIONAL SYSTEMS.
11 11 Planning an ideal architecture is not just about applying new technology. It is also important to understand how to integrate new technology with existing platforms and systems. This includes augmenting what is working already while offloading the realtime performance and streaming Big Data management bottleneck. The core of a real-time enterprise architecture is a streaming data processing platform capable of high volume, high velocity data acquisition, and continuous integration of data across all existing operational siloes. The stream processing platform operates on the arriving sensor data before the data are stored, providing in-memory fast analytics, geospatial analysis, and predictive alerts over the data as it streams past into the offline systems. In summary: Low latency, real-time traffic information. Eliminate latency at all stages - data acquisition, real-time analysis, real-time forecasting, and predictive analytics in order to deliver real-time actionable intelligence that can be utilized by transportation agencies to address congestion and incidents in real-time, and by commuters using smartphone apps for real-tome traffic and route information. Eliminate data siloes through continuous, real-time integration. Streaming integration (continuous ETL) of data from sensors, historic data, applications and databases enables wider visibility and better automation of key operational processes. Combine real-time and historical trend information for optimum real-time decision-making. Databases, data warehouses, and data historians contain mined data and trend information that can be streamed out and joined with the real-time arriving data in order to separate business as usual events from real business exceptions. Collect all data sources. Augment existing information by streaming and joining data of all types, including GPS, Bluetooth, in-road sensors, Twitter and social media, weather data and traveler location information. Solution architecture for real-time operations. It is currently difficult to share data sources across different applications in real-time when the data is held in many different vertical business and operational siloes. A real-time data hub built on a stream processing platform eliminates redundancy through continuous ETL and supports the real-time applications needed for reducing congestion and delivering a better traveler experience.
How To Make Data Streaming A Real Time Intelligence
REAL-TIME OPERATIONAL INTELLIGENCE Competitive advantage from unstructured, high-velocity log and machine Big Data 2 SQLstream: Our s-streaming products unlock the value of high-velocity unstructured log
More informationProcessing and Analyzing Streams. CDRs in Real Time
Processing and Analyzing Streams of CDRs in Real Time Streaming Analytics for CDRs 2 The V of Big Data Velocity means both how fast data is being produced and how fast the data must be processed to meet
More informationAchieving new levels of operational efficiency
SPE-163698-MS Real-Time Streaming Data Management as a Platform for Large-Scale Mission Critical Sensor Network Applications Ronnie Beggs, SQLstream Inc., and Victor H. Abadie III, Consulting Geologist
More informationSAP and Hortonworks Reference Architecture
SAP and Hortonworks Reference Architecture Hortonworks. We Do Hadoop. June Page 1 2014 Hortonworks Inc. 2011 2014. All Rights Reserved A Modern Data Architecture With SAP DATA SYSTEMS APPLICATIO NS Statistical
More informationInternational Journal of Advanced Engineering Research and Applications (IJAERA) ISSN: 2454-2377 Vol. 1, Issue 6, October 2015. Big Data and Hadoop
ISSN: 2454-2377, October 2015 Big Data and Hadoop Simmi Bagga 1 Satinder Kaur 2 1 Assistant Professor, Sant Hira Dass Kanya MahaVidyalaya, Kala Sanghian, Distt Kpt. INDIA E-mail: simmibagga12@gmail.com
More informationSQLstream Blaze and Apache Storm A BENCHMARK COMPARISON
SQLstream Blaze and Apache Storm A BENCHMARK COMPARISON 2 The V of Big Data Velocity means both how fast data is being produced and how fast the data must be processed to meet demand. Gartner The emergence
More informationHow To Handle Big Data With A Data Scientist
III Big Data Technologies Today, new technologies make it possible to realize value from Big Data. Big data technologies can replace highly customized, expensive legacy systems with a standard solution
More informationLuncheon Webinar Series May 13, 2013
Luncheon Webinar Series May 13, 2013 InfoSphere DataStage is Big Data Integration Sponsored By: Presented by : Tony Curcio, InfoSphere Product Management 0 InfoSphere DataStage is Big Data Integration
More informationManaging Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database
Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica
More informationThe 4 Pillars of Technosoft s Big Data Practice
beyond possible Big Use End-user applications Big Analytics Visualisation tools Big Analytical tools Big management systems The 4 Pillars of Technosoft s Big Practice Overview Businesses have long managed
More informationBIG DATA TECHNOLOGY. Hadoop Ecosystem
BIG DATA TECHNOLOGY Hadoop Ecosystem Agenda Background What is Big Data Solution Objective Introduction to Hadoop Hadoop Ecosystem Hybrid EDW Model Predictive Analysis using Hadoop Conclusion What is Big
More informationStreaming Big Data Performance Benchmark. for
Streaming Big Data Performance Benchmark for 2 The V of Big Data Velocity means both how fast data is being produced and how fast the data must be processed to meet demand. Gartner Static Big Data is a
More informationThe big data revolution
The big data revolution Friso van Vollenhoven (Xebia) Enterprise NoSQL Recently, there has been a lot of buzz about the NoSQL movement, a collection of related technologies mostly concerned with storing
More informationUsing Big Data for Smarter Decision Making. Colin White, BI Research July 2011 Sponsored by IBM
Using Big Data for Smarter Decision Making Colin White, BI Research July 2011 Sponsored by IBM USING BIG DATA FOR SMARTER DECISION MAKING To increase competitiveness, 83% of CIOs have visionary plans that
More informationEnd to End Solution to Accelerate Data Warehouse Optimization. Franco Flore Alliance Sales Director - APJ
End to End Solution to Accelerate Data Warehouse Optimization Franco Flore Alliance Sales Director - APJ Big Data Is Driving Key Business Initiatives Increase profitability, innovation, customer satisfaction,
More informationIndustry Impact of Big Data in the Cloud: An IBM Perspective
Industry Impact of Big Data in the Cloud: An IBM Perspective Inhi Cho Suh IBM Software Group, Information Management Vice President, Product Management and Strategy email: inhicho@us.ibm.com twitter: @inhicho
More informationStreaming Big Data Performance Benchmark for Real-time Log Analytics in an Industry Environment
Streaming Big Data Performance Benchmark for Real-time Log Analytics in an Industry Environment SQLstream s-server The Streaming Big Data Engine for Machine Data Intelligence 2 SQLstream proves 15x faster
More informationHow to Enhance Traditional BI Architecture to Leverage Big Data
B I G D ATA How to Enhance Traditional BI Architecture to Leverage Big Data Contents Executive Summary... 1 Traditional BI - DataStack 2.0 Architecture... 2 Benefits of Traditional BI - DataStack 2.0...
More informationSQLstream 4 Product Brief. CHANGING THE ECONOMICS OF BIG DATA SQLstream 4.0 product brief
SQLstream 4 Product Brief CHANGING THE ECONOMICS OF BIG DATA SQLstream 4.0 product brief 2 Latest: The latest release of SQlstream s award winning s-streaming Product Portfolio, SQLstream 4, is changing
More informationAn Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database
An Oracle White Paper June 2012 High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database Executive Overview... 1 Introduction... 1 Oracle Loader for Hadoop... 2 Oracle Direct
More informationINTRODUCTION TO APACHE HADOOP MATTHIAS BRÄGER CERN GS-ASE
INTRODUCTION TO APACHE HADOOP MATTHIAS BRÄGER CERN GS-ASE AGENDA Introduction to Big Data Introduction to Hadoop HDFS file system Map/Reduce framework Hadoop utilities Summary BIG DATA FACTS In what timeframe
More informationANALYTICS BUILT FOR INTERNET OF THINGS
ANALYTICS BUILT FOR INTERNET OF THINGS Big Data Reporting is Out, Actionable Insights are In In recent years, it has become clear that data in itself has little relevance, it is the analysis of it that
More informationReference Architecture, Requirements, Gaps, Roles
Reference Architecture, Requirements, Gaps, Roles The contents of this document are an excerpt from the brainstorming document M0014. The purpose is to show how a detailed Big Data Reference Architecture
More informationUnderstanding the Value of In-Memory in the IT Landscape
February 2012 Understing the Value of In-Memory in Sponsored by QlikView Contents The Many Faces of In-Memory 1 The Meaning of In-Memory 2 The Data Analysis Value Chain Your Goals 3 Mapping Vendors to
More informationDatenverwaltung im Wandel - Building an Enterprise Data Hub with
Datenverwaltung im Wandel - Building an Enterprise Data Hub with Cloudera Bernard Doering Regional Director, Central EMEA, Cloudera Cloudera Your Hadoop Experts Founded 2008, by former employees of Employees
More informationCybersecurity Analytics for a Smarter Planet
IBM Institute for Advanced Security December 2010 White Paper Cybersecurity Analytics for a Smarter Planet Enabling complex analytics with ultra-low latencies on cybersecurity data in motion 2 Cybersecurity
More information5 Keys to Unlocking the Big Data Analytics Puzzle. Anurag Tandon Director, Product Marketing March 26, 2014
5 Keys to Unlocking the Big Data Analytics Puzzle Anurag Tandon Director, Product Marketing March 26, 2014 1 A Little About Us A global footprint. A proven innovator. A leader in enterprise analytics for
More informationWhy Big Data in the Cloud?
Have 40 Why Big Data in the Cloud? Colin White, BI Research January 2014 Sponsored by Treasure Data TABLE OF CONTENTS Introduction The Importance of Big Data The Role of Cloud Computing Using Big Data
More informationTRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS
9 8 TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS Assist. Prof. Latinka Todoranova Econ Lit C 810 Information technology is a highly dynamic field of research. As part of it, business intelligence
More informationNative Connectivity to Big Data Sources in MicroStrategy 10. Presented by: Raja Ganapathy
Native Connectivity to Big Data Sources in MicroStrategy 10 Presented by: Raja Ganapathy Agenda MicroStrategy supports several data sources, including Hadoop Why Hadoop? How does MicroStrategy Analytics
More informationBIG DATA CHALLENGES AND PERSPECTIVES
BIG DATA CHALLENGES AND PERSPECTIVES Meenakshi Sharma 1, Keshav Kishore 2 1 Student of Master of Technology, 2 Head of Department, Department of Computer Science and Engineering, A P Goyal Shimla University,
More informationW H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract
W H I T E P A P E R Deriving Intelligence from Large Data Using Hadoop and Applying Analytics Abstract This white paper is focused on discussing the challenges facing large scale data processing and the
More informationTHE DEVELOPER GUIDE TO BUILDING STREAMING DATA APPLICATIONS
THE DEVELOPER GUIDE TO BUILDING STREAMING DATA APPLICATIONS WHITE PAPER Successfully writing Fast Data applications to manage data generated from mobile, smart devices and social interactions, and the
More informationFrom Spark to Ignition:
From Spark to Ignition: Fueling Your Business on Real-Time Analytics Eric Frenkiel, MemSQL CEO June 29, 2015 San Francisco, CA What s in Store For This Presentation? 1. MemSQL: A real-time database for
More informationThe Next Wave of Data Management. Is Big Data The New Normal?
The Next Wave of Data Management Is Big Data The New Normal? Table of Contents Introduction 3 Separating Reality and Hype 3 Why Are Firms Making IT Investments In Big Data? 4 Trends In Data Management
More informationAlexander Nikov. 5. Database Systems and Managing Data Resources. Learning Objectives. RR Donnelley Tries to Master Its Data
INFO 1500 Introduction to IT Fundamentals 5. Database Systems and Managing Data Resources Learning Objectives 1. Describe how the problems of managing data resources in a traditional file environment are
More informationData Refinery with Big Data Aspects
International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 3, Number 7 (2013), pp. 655-662 International Research Publications House http://www. irphouse.com /ijict.htm Data
More informationTesting Big data is one of the biggest
Infosys Labs Briefings VOL 11 NO 1 2013 Big Data: Testing Approach to Overcome Quality Challenges By Mahesh Gudipati, Shanthi Rao, Naju D. Mohan and Naveen Kumar Gajja Validate data quality by employing
More informationIBM Analytics The fluid data layer: The future of data management
IBM Analytics The fluid data layer: The future of data management Why flexibility and adaptability are crucial in the hybrid cloud world 1 2 3 4 5 6 The new world vision for data architects Why the fluid
More informationBIG DATA & ANALYTICS. Transforming the business and driving revenue through big data and analytics
BIG DATA & ANALYTICS Transforming the business and driving revenue through big data and analytics Collection, storage and extraction of business value from data generated from a variety of sources are
More informationExploiting Data at Rest and Data in Motion with a Big Data Platform
Exploiting Data at Rest and Data in Motion with a Big Data Platform Sarah Brader, sarah_brader@uk.ibm.com What is Big Data? Where does it come from? 12+ TBs of tweet data every day 30 billion RFID tags
More informationBig Data Analytics. Copyright 2011 EMC Corporation. All rights reserved.
Big Data Analytics 1 Priority Discussion Topics What are the most compelling business drivers behind big data analytics? Do you have or expect to have data scientists on your staff, and what will be their
More informationArchitecting an Industrial Sensor Data Platform for Big Data Analytics: Continued
Architecting an Industrial Sensor Data Platform for Big Data Analytics: Continued 2 8 10 Issue 1 Welcome From the Gartner Files: Blueprint for Architecting Sensor Data for Big Data Analytics About OSIsoft,
More informationNextGen Infrastructure for Big DATA Analytics.
NextGen Infrastructure for Big DATA Analytics. So What is Big Data? Data that exceeds the processing capacity of conven4onal database systems. The data is too big, moves too fast, or doesn t fit the structures
More informationHadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012. Viswa Sharma Solutions Architect Tata Consultancy Services
Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012 Viswa Sharma Solutions Architect Tata Consultancy Services 1 Agenda What is Hadoop Why Hadoop? The Net Generation is here Sizing the
More informationCisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database
Cisco UCS and Fusion- io take Big Data workloads to extreme performance in a small footprint: A case study with Oracle NoSQL database Built up on Cisco s big data common platform architecture (CPA), a
More informationDeploying Big Data to the Cloud: Roadmap for Success
Deploying Big Data to the Cloud: Roadmap for Success James Kobielus Chair, CSCC Big Data in the Cloud Working Group IBM Big Data Evangelist. IBM Data Magazine, Editor-in- Chief. IBM Senior Program Director,
More informationSoftware AG Fast Big Data Solutions. Come la gestione realtime dei dati abilita nuovi scenari di business per le Banche
Software AG Fast Big Data Solutions Come la gestione realtime dei dati abilita nuovi scenari di business per le Banche Software AG Fast Big Data Solutions Get there faster Vittorio Carosone Regional Sales
More informationManifest for Big Data Pig, Hive & Jaql
Manifest for Big Data Pig, Hive & Jaql Ajay Chotrani, Priyanka Punjabi, Prachi Ratnani, Rupali Hande Final Year Student, Dept. of Computer Engineering, V.E.S.I.T, Mumbai, India Faculty, Computer Engineering,
More informationWHITE PAPER SPLUNK SOFTWARE AS A SIEM
SPLUNK SOFTWARE AS A SIEM Improve your security posture by using Splunk as your SIEM HIGHLIGHTS Splunk software can be used to operate security operations centers (SOC) of any size (large, med, small)
More informationWhy Big Data Analytics?
An ebook by Datameer Why Big Data Analytics? Three Business Challenges Best Addressed Using Big Data Analytics It s hard to overstate the importance of data for businesses today. It s the lifeline of any
More informationOffload Enterprise Data Warehouse (EDW) to Big Data Lake. Ample White Paper
Offload Enterprise Data Warehouse (EDW) to Big Data Lake Oracle Exadata, Teradata, Netezza and SQL Server Ample White Paper EDW (Enterprise Data Warehouse) Offloads The EDW (Enterprise Data Warehouse)
More informationBig Data Are You Ready? Jorge Plascencia Solution Architect Manager
Big Data Are You Ready? Jorge Plascencia Solution Architect Manager Big Data: The Datafication Of Everything Thoughts Devices Processes Thoughts Things Processes Run the Business Organize data to do something
More informationArchitecting for Big Data Analytics and Beyond: A New Framework for Business Intelligence and Data Warehousing
Architecting for Big Data Analytics and Beyond: A New Framework for Business Intelligence and Data Warehousing Wayne W. Eckerson Director of Research, TechTarget Founder, BI Leadership Forum Business Analytics
More informationIntelligent Business Operations and Big Data. 2014 Software AG. All rights reserved.
Intelligent Business Operations and Big Data 1 What is Big Data? Big data is a popular term used to acknowledge the exponential growth, availability and use of information in the data-rich landscape of
More informationGigaSpaces Real-Time Analytics for Big Data
GigaSpaces Real-Time Analytics for Big Data GigaSpaces makes it easy to build and deploy large-scale real-time analytics systems Rapidly increasing use of large-scale and location-aware social media and
More informationSmarter wireless networks
IBM Software Telecommunications Thought Leadership White Paper Smarter wireless networks Add intelligence to the mobile network edge 2 Smarter wireless networks Contents 2 Introduction 3 Intelligent base
More informationTransforming the Telecoms Business using Big Data and Analytics
Transforming the Telecoms Business using Big Data and Analytics Event: ICT Forum for HR Professionals Venue: Meikles Hotel, Harare, Zimbabwe Date: 19 th 21 st August 2015 AFRALTI 1 Objectives Describe
More informationThe Future of Data Management
The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah (@awadallah) Cofounder and CTO Cloudera Snapshot Founded 2008, by former employees of Employees Today ~ 800 World Class
More informationWhite Paper. Version 1.2 May 2015 RAID Incorporated
White Paper Version 1.2 May 2015 RAID Incorporated Introduction The abundance of Big Data, structured, partially-structured and unstructured massive datasets, which are too large to be processed effectively
More informationThe 3 questions to ask yourself about BIG DATA
The 3 questions to ask yourself about BIG DATA Do you have a big data problem? Companies looking to tackle big data problems are embarking on a journey that is full of hype, buzz, confusion, and misinformation.
More informationAssociate Professor, Department of CSE, Shri Vishnu Engineering College for Women, Andhra Pradesh, India 2
Volume 6, Issue 3, March 2016 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Special Issue
More informationBig Data, Why All the Buzz? (Abridged) Anita Luthra, February 20, 2014
Big Data, Why All the Buzz? (Abridged) Anita Luthra, February 20, 2014 Defining Big Not Just Massive Data Big data refers to data sets whose size is beyond the ability of typical database software tools
More informationSQLSaturday #399 Sacramento 25 July, 2015. Big Data Analytics with Excel
SQLSaturday #399 Sacramento 25 July, 2015 Big Data Analytics with Excel Presenter Introduction Peter Myers Independent BI Expert Bitwise Solutions BBus, SQL Server MCSE, SQL Server MVP since 2007 Experienced
More informationForecast of Big Data Trends. Assoc. Prof. Dr. Thanachart Numnonda Executive Director IMC Institute 3 September 2014
Forecast of Big Data Trends Assoc. Prof. Dr. Thanachart Numnonda Executive Director IMC Institute 3 September 2014 Big Data transforms Business 2 Data created every minute Source http://mashable.com/2012/06/22/data-created-every-minute/
More informationBig Data Integration: A Buyer's Guide
SEPTEMBER 2013 Buyer s Guide to Big Data Integration Sponsored by Contents Introduction 1 Challenges of Big Data Integration: New and Old 1 What You Need for Big Data Integration 3 Preferred Technology
More informationBringing Big Data into the Enterprise
Bringing Big Data into the Enterprise Overview When evaluating Big Data applications in enterprise computing, one often-asked question is how does Big Data compare to the Enterprise Data Warehouse (EDW)?
More informationBENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next
More informationBigMemory & Hybris : Working together to improve the e-commerce customer experience
& Hybris : Working together to improve the e-commerce customer experience TABLE OF CONTENTS 1 Introduction 1 Why in-memory? 2 Why is in-memory Important for an e-commerce environment? 2 Why? 3 How does
More informationAligning Your Strategic Initiatives with a Realistic Big Data Analytics Roadmap
Aligning Your Strategic Initiatives with a Realistic Big Data Analytics Roadmap 3 key strategic advantages, and a realistic roadmap for what you really need, and when 2012, Cognizant Topics to be discussed
More informationReimagining Business with SAP HANA Cloud Platform for the Internet of Things
SAP Brief SAP HANA SAP HANA Cloud Platform for the Internet of Things Objectives Reimagining Business with SAP HANA Cloud Platform for the Internet of Things Connect, transform, and reimagine Connect,
More informationHadoop and Data Warehouse Friends, Enemies or Profiteers? What about Real Time?
Hadoop and Data Warehouse Friends, Enemies or Profiteers? What about Real Time? Kai Wähner kwaehner@tibco.com @KaiWaehner www.kai-waehner.de Disclaimer! These opinions are my own and do not necessarily
More informationlocuz.com Big Data Services
locuz.com Big Data Services Big Data At Locuz, we help the enterprise move from being a data-limited to a data-driven one, thereby enabling smarter, faster decisions that result in better business outcome.
More informationBIG DATA THE NEW OPPORTUNITY
Feature Biswajit Mohapatra is an IBM Certified Consultant and a global integrated delivery leader for IBM s AMS business application modernization (BAM) practice. He is IBM India s competency head for
More informationStreaming Analytics and the Internet of Things: Transportation and Logistics
Streaming Analytics and the Internet of Things: Transportation and Logistics FOOD WASTE AND THE IoT According to the Food and Agriculture Organization of the United Nations, every year about a third of
More informationOracle Data Integrator 12c (ODI12c) - Powering Big Data and Real-Time Business Analytics. An Oracle White Paper October 2013
An Oracle White Paper October 2013 Oracle Data Integrator 12c (ODI12c) - Powering Big Data and Real-Time Business Analytics Introduction: The value of analytics is so widely recognized today that all mid
More informationWelcome. Host: Eric Kavanagh. eric.kavanagh@bloorgroup.com. The Briefing Room. Twitter Tag: #briefr
The Briefing Room Welcome Host: Eric Kavanagh eric.kavanagh@bloorgroup.com Twitter Tag: #briefr The Briefing Room Mission! Reveal the essential characteristics of enterprise software, good and bad! Provide
More informationTraditional BI vs. Business Data Lake A comparison
Traditional BI vs. Business Data Lake A comparison The need for new thinking around data storage and analysis Traditional Business Intelligence (BI) systems provide various levels and kinds of analyses
More informationFujitsu Big Data Software Use Cases
Fujitsu Big Data Software Use s Using Big Data Opens the Door to New Business Areas The use of Big Data is needed in order to discover trends and predictions, hidden in data generated over the course of
More informationBig Data and Apache Hadoop Adoption:
Expert Reference Series of White Papers Big Data and Apache Hadoop Adoption: Key Challenges and Rewards 1-800-COURSES www.globalknowledge.com Big Data and Apache Hadoop Adoption: Key Challenges and Rewards
More informationCIO Guide How to Use Hadoop with Your SAP Software Landscape
SAP Solutions CIO Guide How to Use with Your SAP Software Landscape February 2013 Table of Contents 3 Executive Summary 4 Introduction and Scope 6 Big Data: A Definition A Conventional Disk-Based RDBMs
More informationManaging Cloud Server with Big Data for Small, Medium Enterprises: Issues and Challenges
Managing Cloud Server with Big Data for Small, Medium Enterprises: Issues and Challenges Prerita Gupta Research Scholar, DAV College, Chandigarh Dr. Harmunish Taneja Department of Computer Science and
More informationThe Lab and The Factory
The Lab and The Factory Architecting for Big Data Management April Reeve DAMA Wisconsin March 11 2014 1 A good speech should be like a woman's skirt: long enough to cover the subject and short enough to
More informationBIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON
BIG DATA IN THE CLOUD : CHALLENGES AND OPPORTUNITIES MARY- JANE SULE & PROF. MAOZHEN LI BRUNEL UNIVERSITY, LONDON Overview * Introduction * Multiple faces of Big Data * Challenges of Big Data * Cloud Computing
More informationBoarding to Big data
Database Systems Journal vol. VI, no. 4/2015 11 Boarding to Big data Oana Claudia BRATOSIN University of Economic Studies, Bucharest, Romania oc.bratosin@gmail.com Today Big data is an emerging topic,
More informationKeywords Big Data, NoSQL, Relational Databases, Decision Making using Big Data, Hadoop
Volume 4, Issue 1, January 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Transitioning
More informationChapter 6 8/12/2015. Foundations of Business Intelligence: Databases and Information Management. Problem:
Foundations of Business Intelligence: Databases and Information Management VIDEO CASES Chapter 6 Case 1a: City of Dubuque Uses Cloud Computing and Sensors to Build a Smarter, Sustainable City Case 1b:
More informationWhere is... How do I get to...
Big Data, Fast Data, Spatial Data Making Sense of Location Data in a Smart City Hans Viehmann Product Manager EMEA ORACLE Corporation August 19, 2015 Copyright 2014, Oracle and/or its affiliates. All rights
More informationSpeeding ETL Processing in Data Warehouses White Paper
Speeding ETL Processing in Data Warehouses White Paper 020607dmxwpADM High-Performance Aggregations and Joins for Faster Data Warehouse Processing Data Processing Challenges... 1 Joins and Aggregates are
More informationTIBCO Live Datamart: Push-Based Real-Time Analytics
TIBCO Live Datamart: Push-Based Real-Time Analytics ABSTRACT TIBCO Live Datamart is a new approach to real-time analytics and data warehousing for environments where large volumes of data require a management
More informationWhat happens when Big Data and Master Data come together?
What happens when Big Data and Master Data come together? Jeremy Pritchard Master Data Management fgdd 1 What is Master Data? Master data is data that is shared by multiple computer systems. The Information
More informationDecoding the Big Data Deluge a Virtual Approach. Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco
Decoding the Big Data Deluge a Virtual Approach Dan Luongo, Global Lead, Field Solution Engineering Data Virtualization Business Unit, Cisco High-volume, velocity and variety information assets that demand
More informationThe IBM Cognos Platform
The IBM Cognos Platform Deliver complete, consistent, timely information to all your users, with cost-effective scale Highlights Reach all your information reliably and quickly Deliver a complete, consistent
More informationDelivering secure, real-time business insights for the Industrial world
Delivering secure, real-time business insights for the Industrial world Arnaud Mathieu: Program Director, Internet of Things Dev., IBM amathieu@us.ibm.com @arnomath 1 We are on the threshold of massive
More informationAn Implementation of Active Data Technology
White Paper by: Mario Morfin, PhD Terri Chu, MEng Stephen Chen, PhD Robby Burko, PhD Riad Hartani, PhD An Implementation of Active Data Technology October 2015 In this paper, we build the rationale for
More informationBig Data at Cloud Scale
Big Data at Cloud Scale Pushing the limits of flexible & powerful analytics Copyright 2015 Pentaho Corporation. Redistribution permitted. All trademarks are the property of their respective owners. For
More informationHow the oil and gas industry can gain value from Big Data?
How the oil and gas industry can gain value from Big Data? Arild Kristensen Nordic Sales Manager, Big Data Analytics arild.kristensen@no.ibm.com, tlf. +4790532591 April 25, 2013 2013 IBM Corporation Dilbert
More informationThe Internet of Things and Big Data: Intro
The Internet of Things and Big Data: Intro John Berns, Solutions Architect, APAC - MapR Technologies April 22 nd, 2014 1 What This Is; What This Is Not It s not specific to IoT It s not about any specific
More informationBig Data Are You Ready? Thomas Kyte http://asktom.oracle.com
Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated
More information