SQL 2012: Bringing Big Data to the Desktop

Transcription

1 Big data is getting bigger all the time. According to Gartner, the volume of data worldwide is increasing by at least 59% every year and 85% is unstructured data from a growing variety of new sources. The job for the future is to meet the challenges of storing, processing, and interpreting this new data, and ultimately, converting it to the insight needed to drive growth. New technologies and tools in Microsoft SQL Server 2012 will help businesses meet these challenges. SQL 2012 is a unified solution to maximize the impact of big data from the point of collection, to presentation-ready reports. It empowers the end user with extraordinary tools for insight, and helps decision-makers control IT costs while gaining insight at the speed of business. Companies that have begun to exploit the potential of big data have already achieved a tremendous business intelligence (BI) advantage over competitors that have yet to engage with its emerging capabilities. Another commonly cited statistic from Gartner is that by 2015, 85% of all companies will fail to make effective use of their data. According to the Aberdeen Group, the top performing companies are the ones who have also achieved the greatest level of complete and accurate data. Big data is % of Master Complete and Accurate 6 Under 50% Bottom 30% 68% 2.8 Middle 50% 91% 1.2 Top 20% Figure 1. Credible Consistent Hours Spent per Employee Each Week Searching for Information Digital video GPS information from RFID tag readers from smart meters Smartphone data Social media and sentiment analytics Affinity tracking Log analysis Medical records and images Click data for ad targeting Financial analysis Companies are beginning to hold on to data that they previously let slip through their fingers because processing it was resource intensive. This has led to an explosion in the volume and variety of data, and the resulting problems with capacity. Beyond that, new tools for visualizing data and bringing insight to stakeholders and decision makers are revolutionizing the role of data. Gathering & Structuring Big Agility and responsiveness to data are just as important as the capacity to collect, store and process it. Traditional technologies get in the way of this responsiveness, as data visualization, reporting, and analysis are dependent on IT for these functions. When BI wants new analytics, the impact and value of new insight can be blunted by costly delays while IT designs and implements a solution. more than massive databases. Increasingly, it is rich content that comes from entirely new sources, and in a multitude of forms. To name a few: 1 magenic.com

2 Third-Party BI Apps Reporting Services Reports Excel Workbooks PowerPivot Apps SharePoint Dashboards & Scorecards BI Applications Relational DAX VertiPaq Multidimensional MDX Real-time Model Business Logic Access SQL Server Oracle SAP Microsoft Dynamics As with most technologies, there are potential roadblocks to adoption such as cost, capacity, access, security, interpretation, and visualization. SQL 2012 has added new capabilities that deliver faster speed to insight, scalability, and command of more kinds of data from more data sources; all in a unified solution. Microsoft has integrated with Apache Hadoop to process enormous quantities of both structured and unstructured data from a wide variety of sources. Hadoop can scrape data from any source in any form, as well as manage the storage and large-scale number crunching demanded by big data. Hadoop is designed to easily scale up from a single server to thousands of machines and will also be supported in Azure. This provides opportunities for scale-on-demand solutions with seamless and secure data communications between the client and the cloud. Hadoop s MapReduce framework allows developers to easily and quickly develop and test applications, saving time and money. The framework s design minimizes the impact of hardware failure of individual nodes, data congestion, and other big data pitfalls by distributing the work across a number of machines. The Hadoop tutorial on yahoo.com describes this as: moving the computation to the data, instead of moving the data to the computation. The result is much higher performance. Files Figure 2. BI Semantics Model Odata Feeds Sources This speed and efficiency is demonstrated in a case study of a project by the data services company, Klout. According to a case study published by Microsoft in June 2012, Klout tracks hundreds of terabytes of data every day on the 15 biggest social networking sites, and help their customers understand and manage their influence on the web. They needed to increase the speed and efficiency of mission- and business-critical BI systems. In 2011, Klout implemented a BI solution using SQL 2012 in conjunction with Hadoop and Hive. David Mariani Klout Vice President of Engineering says, We can [now] use SQL Server 2012 Analysis Services and Microsoft BI tools to perform ad-hoc queries of big data on Hadoop at sub-second response times. This faster response time comes with a greater efficiency and lower cost in implementing and maintaining solutions. Mariani says, By using open-source and commercial software, we don t have to build everything from the ground up and we get a great ecosystem of tools and support. Finally, the flexibility and flat scalability offered by this combination of Hive, Hadoop, and SQL 2012 will facilitate innovation: When it comes to business intelligence, Microsoft SQL Server 2012 demonstrates that the platform has continued 2 magenic.com

3 _Listener Synchronous Movement New Jersey Hong Kong New York (Primary) Asynchronous Movement Toronto to advance and keep up with the innovations that are happening in big data, says Mariani. We re very excited about working with Microsoft to develop solutions based on technologies like SQL Server PowerPivot for Excel and Hadoop so that we can continue to deliver unique services that transform what is possible with big data. (Microsoft, Klout ) Big High Availbility Another source of high performance with SQL 2012 is AlwaysOn. AlwaysOn is a high-availability solution that improves on previous strategies, such as database mirroring and clustering. With AlwaysOn, multiple active secondary copies of databases provide greater availability, data security, and disaster recovery. Downtime and data protection concerns have reached a very high level of refinement, and when up-to-the-minute data and rock-solid reliability are absolute needs, every improvement is welcomed. AlwaysOn allows users near real-time access to data, while maintaining a consistent level of performance for those entering the data. This is one way that SQL Server gets IT out of the way of business. producers and consumers will no longer fight for resources and access to do their jobs. AlwaysOn creates one copy of enterprise data that can be updated, and up to four active secondary copies to be consumed by users. This redundancy allows a higher level of data protection, as well as fail-over and recovery of data should a problem arise with one of the copies. This greatly reduces the risk of downtime and data loss, and Figure 3. Greater Uptime with AlwaysOn Barcelona reduces costs. Some dramatic increases in performance have been experienced by early adopters implementing SQL 2012 to take advantage of AlwaysOn. According to a Microsoft case study, one of China s largest software developers, Yonyou, experienced a nearly 70 percent increase in the performance of their ERP application: In its testing of SQL Server 2012 AlwaysOn, Yonyou saw the number of transactions per second increase from 4.8 to 8.2, with an improvement of 70 percent and system throughput improved substantially. Based on a comparison of Average Transaction Response Times for both business transactions and reporting scenarios, Yonyou observed a percent improvement. (Microsoft, Yonyou Software Co. Ltd. ) ColumnStore Index is another feature which increases the speed of some queries, sometimes by a factor of 10 to 100 in large-memory, multi-core systems. The data is organized and compressed for faster returns, even with very large amounts of data. (Microsoft Technet Wiki) This means that businesses can get better performance with less hardware, which lowers TCO. These additions greatly increase capacity, security, variety, and speed of access to the mass of data that forms the base of great BI. Putting Big to Work on the Desktop SQL 2012 eliminates the traditional need to engage IT for the creation of reporting and visualization with PowerPivot and Power View. These two tools drive decision-making to the desktop, allowing the end user to create their own ad-hoc reports, charts, and presentation-ready visualizations. PowerPivot creates workbooks with cross-tool compatibility. 3 magenic.com

4 Workbooks created in one tool can be shared and passed to any of the other Microsoft BI tools, including PowerPoint, Developer Visual Studio, Word, or Power View. This is because of the Business Intelligence Semantic Model (BISM) integrated into the platform of SQL The BISM is one model for all end user experiences. This, and a powerful new tool set for faster development, allows IT to construct the data model, the underlying structure through which the data can be accessed. They plan for the security and fidelity of the data, as well as tailoring access to and visualization of data for various roles. These efforts are all properly the responsibility of IT, and are an effective use of the resources that they represent. Combined with Power View, they give the end user intuitive command over the analysis and visualization of data. Power View puts the power of data visualization at the user s fingertips. Magenic principal consultant Steve Hughes calls Power View a game changer for the end user. The user is given access to the data through a series of telescoping menus, designed by an IT professional familiar with the user s requirements. The user interface and menu are designed to anticipate the user s needs without limiting the user to a rigid set of standard reports. The user selects data for their own analytics with drag-and-drop simplicity to create ad-hoc reports and charts. What would previously require weeks of waiting for IT to design and create a solution can now be done at the desktop, as quickly as the user can work the mouse. The user can also select from rules provided by the data model, and create report alerts to continuously monitor critical data and inform them when certain conditions are met. Microsoft partner and blogger Simran Jindal said in her blog: Power View is definitely a great complement to the SharePoint BI tool set making it possible to quickly visualize the data in a fun Microsoft SQL Server On-premises Hosted/ Public Cloud Private Cloud/ Hybrid Cloud Figure 4. Enables Scale on Demand and interactive way. To me it provides a canvas for the creative mind to go wild with the data presentation capabilities. The importance of this UX focus is driven home when considering the underutilization of business intelligence solutions in many businesses. The tools are only useful if the users make the most of them. Faster Development, Testing & Maintenance This virtuosity of self-service BI and data visualization in enterprise systems is dependent on a firm foundation of IT data modeling and UX design. The tools provided to IT for the design of the UX are an important part of the value proposition of SQL Developers in SQL 2012 are able to take full advantage of recently expanded capabilities in SSIS (SQL Server Integration Services) and SSDT (SQL Server Tools). Some results of these improvements are: Faster development with fewer mistakes SQL Azure becomes a first class citizen Cleaner integration between DBAs and developers, including production deployment These capabilities reduce the amount of time and money needed to develop, test, and maintain new applications for SQL Server and SQL Azure. (Barskiy) In addition to increased utilization of tools and increased timeliness of insight, companies that use the BI solutions available to them through SQL 2012 save on IT resources. According to a case study by Microsoft, the IT services firm SharePoint 360 was able to save an estimated $268,000 annually in labor costs by empowering their staff with self-service BI tools. This enabled them to reduce the amount of time they needed to dedicate consultants to 4 magenic.com

5 internal BI projects, rather than to customer projects. (Microsoft, SharePoint 360 ) Innovations in SQL 2012 have resulted in: Greater command of a wider variety of data Increased speed to insight Enhanced data availability and protection Reduced development and maintenance time New tools for Businesses to control IT costs Users empowered with amazing new self-service BI capabilities that will enhance decision-making and drive growth SQL 2012 provides a unified solution that creates faster speed to insight from collection to presentation. Implementation of SQL 2012 in both business and IT environments can be done seamlessly, securely, and efficiently by a team of professionals with the skills and experience to design custom solutions that balance the needs of the customer, their current business and IT environments, and budgetary concerns. Only 28% of potential users are using the standard Business Intelligence platform of choice in organizations 1 Resources Aberdeen Group. Turning Pain into Productivity with Master Management. Feb, Barskiy, Sergey. Principal Consultant South Region, Magenic. Interview 4/30/2012 Beyer, Mark. Gartner Symposium Presentation Information Management Goes Extreme : The biggest Challenges for 21st- Century CIOs. Oct Hughes, Steve. on Wheels Why I am Excited About SQL Server why-i-am-excited-about-sql-server-2012/ Jindal, Simran. Simran Jindal s Blog: PowerView How Does it Affect the Current Landscape of SharePoint Reporting? Edge-and-Delivers-More-Innovative-BI-in-Less-Time/ Microsoft Corporation. Case Study: Yonyou Software Co. Ltd. May 23, Boost/ Microsoft Corporation. Case Study: Klout. June 3, Hadoop-to-Boost-Insight-into-Big-/ Microsoft Corporation. SQL Server Team Blog. simranjindal.wordpress.com/2012/02/15/powerview-how-does-itaffect-the-current-landscape-of-sharepoint-reporting/ Microsoft Corporation. SQL: Server 2012 Partner Event Presentation. March 6, Microsoft Corporation. Case Study: Sharepoint 360. Jan. 26, Server-2012/SharePoint360/Consulting-Firm-Gains-Competitive- Microsoft-SQL-Server-2012/Yonyou-Software-Co.-Ltd/Software- Maker-Upgrades-base-to-Achieve-70-Percent-Performance- Enterprise/Klout/-Services-Firm-Uses-Microsoft-BI-and- technet.com/b/dataplatforminsider/archive/2010/11/12/analysisservices-roadmap-for-sql-server-denali-and-beyond.aspx Microsoft Technet Wiki. SQL Server Columnstore Index FAQ (en-us) articles/3540.sql-server-columnstore-index-faq-en-us. aspx#overview Yahoo Developer Network Module 1: Tutorial, The Hadoop Approach, Distribution tutorial/module1.html#data Engage Magenic today online at magenic.com or by calling our sales line at magenic.com