Effective Data Integration - where to begin. Bryte Systems

Transcription

1 Effective Data Integration - where to begin Bryte Systems

2 making data work Bryte Systems specialises is providing innovative and cutting-edge data integration and data access solutions and products to enable a data savvy enterprise. Bryte Systems offers products for access and integration of Big Data, Change Data Capture, Data Integration, Data Replication and Metadata Management. Contact us: Website: Call us: Telephone: Fax: Address: Level 20, Tower A The Zenith Centre 821 Pacific Highway Chatswood NSW 2067 Australia us: info@bryte.com.au

3 How Data Integration is evolving Data Integration technology is here to stay. However, the technology faces new challenges now, with the rise of cloud computing, the emerging use of big data and the need to use data integration tools across a range of different underlying architectures and data technologies. Data is critical to running a business, small or large, but organisations cannot realize the full business value of their data unless they can effectively integrate and move it between different systems regardless of where it originates, in the enterprise or on the Internet. And now the need for Data Integration technologies becomes even more critical, as the tools are required to handle the traditional data warehouses and unstructured sources - across the internet, from different appliances, files, Hadoop, just to name a few and integrate them seamlessly to deliver the strategic business advantage. Emerging technologies such as data tiering and compression, along with in-memory databases, are enabling the management of much larger workloads than conventional warehouses. The reasons for requiring data integration still stand as they did in the past for delivering Single source of truth Single view of customer Reduction in data acquisition costs Current and historical data for trend analysis Data consolidation for reporting and analytics Data lineage But now, old slow moving Data Warehousing solutions delivering to Business Intelligence applications can be better implemented with the new technologies. Further the typical payback for investment in the new technologies is just a few months, as the tools are more costeffective and data delivery is efficient and streamlined. Bryte Systems: Effective Data Integration where to begin Copyright 2013 All rights reserved Page 3

4 The movement to big data systems these days is largely driven around the commoditisation of technology, and the availability of cheap and massively parallel and scalable platforms, such as those provided by cloud computing providers. Another factor is the ability to manage data at speeds once not considered possible, given the ability of systems such as Hadoop which provide reliable data storage and high-performance parallel data processing. Sourcing unstructured data from machines or the internet is becoming necessary for gaining a competitive advantage. Data Integration now needs to scale up for: Data Flow Volume an exponential increase with certain sources Structured Data Unstructured Data Moving data effectively between the cloud and enterprise data repositories Delivering accurate data real-time or near real-time Built-in processes and techniques for data manipulations and transformations so that complexity is hidden and productivity is boosted More and more organisations are realizing that data integration done right can be a key differentiator. But, with the ever-changing integration landscape, how can IT professionals be sure they are taking the right approach? What constitutes an effective data integration technology?

5 Effective Data Integration An effective data integration technology has to deliver on the following objectives: Timeliness Data is a perishable commodity: the older it is, the less relevant. Businesses need tools that can provide real-time data flowing into their business intelligence applications for a current and comprehensive picture of their organisation and their customers. Accuracy and Integrity Business intelligence based on wrong data will do more harm than good. The data integration environment should drive a higher quality of audit and error mechanisms to include automatic or convenient error checking and verification. Consistency and Clarity Integration should provide a clear and unique definition of various types of data, and promote a better alignment in the organisation so that everyone is working from the single source of truth. Completeness All available information, including both historical and recent data, should be made accessible in an integrated database, with any missing records or fields identified and flagged via the integration process. Quick Turn-around The integrated data environment should be a consolidated environment that provides data that has been pre-processed, consolidated and verified for various applications to use concurrently. Productivity The data integration technology should provide developers with flexibility, ease of use and reusability. Known techniques should be provided as built-in processes so that delivery and maintenance is further simplified Bryte Systems: Effective Data Integration where to begin Copyright 2013 All rights reserved Page 5

6 Scalability With the emergence of big data, scalability is even more vital as big data brings with it a deluge of data. Data integration should scale and be extensible when you need it. Versatility The data integration technology should be able to deal with different underlying architectures and technologies as it would need to cope with structured and unstructured data environments. Lower Data Acquisition and Storage Cost Data integration should allow data to be collected only once, and made readily available to all users. Security Data security and especially data privacy which gets more relevant with big data is a critical part of information management, and must be managed effectively by the integration technology.

7 Conclusion As data volumes, variety and complexity continue to soar; traditional data integration technologies face a battle on their hands to keep up with ever-increasing and changing demands of the business world. Organisations need to consider their data integration technology carefully, as many are still stuck in the past and have not realised just how much better data integration software has become in the recent years. The slow moving data from ETL products with proprietary servers that handled data row by row and the slow processing of data are a thing of the past. New data integration technologies bring real-time data delivery, better data management techniques that remove the need for lengthy batch processing, better error handling and enable turn-key solutions that can be more responsive to business demands. Organisations also need to clearly understand the type and volume of data that they need to support, now and into the future. This due diligence should ensure that the integration tool they acquire today will continue to deliver value tomorrow and well into the future. Effective Data integration continues to be an important piece of the enterprise technology puzzle. As time moves on, it will become even more critical for success. Bryte Systems: Effective Data Integration where to begin Copyright 2013 All rights reserved Page 7