Big Data for Law Firms DAMIAN BLACKBURN

Transcription

1 DAMIAN BLACKBURN PUBLISHED BY IN IN ASSOCIATION WITH

2 is published by Ark Group UK/EUROPE/ASIA OFFICE Ark Group Ltd 6-14 Underwood Street London N1 7JQ United Kingdom Tel +44 (0) Fax +44 (0) NORTH AMERICA OFFICE Ark Group USA 4408 N. Rockwood Drive Suite 150 Peoria IL United States Tel Fax AUSTRALIA/NZ OFFICE Ark Group Australia Pty Ltd Main Level 83 Walker Street North Sydney NSW 2060 Australia Tel Fax Online bookshop Reports Commissioning Editor Helen Roche Reports Publisher International Fiona Tucker UK/Europe/Asia enquiries Robyn Macé US enquiries Daniel Smallwood Australia/NZ enquiries Steve Oesterreich ISBN: (hard copy) (PDF) Copyright The copyright of all material appearing within this publication is reserved by the author and Ark Conferences It may not be reproduced, duplicated or copied by any means without the prior written consent of the publisher. ARK2369

3 Chapter 1: Introduction to big data SINCE THE birth of computing and the accompanying issue of data storage, individuals and institutions of all sizes have been collecting, storing, manipulating, and repurposing data. For the majority of its life, electronically stored data has been organised into flat file and relational databases, which are relatively straightforward repositories. Rules and methods for storing and accessing this data were formed and were largely compliant across the majority of various types of commercial and open source database systems. In essence, a standard for data storage and manipulation was formed, and that standard applied to any size of repository. This situation lasted for a good number of years until very recently, when thought resources were diverted to a new and necessary form of data system: big data. During the last few years, there has been an exponential increase in the amount of data generated and stored around the globe. The increase in quantity and the increasing variety of types of data generated meant that conventional systems became overwhelmed. Companies generating, storing, and manipulating these vast silos of data had to develop new tools and techniques in order to be able to extract value from them, and this is collectively known as big data. Data is measured in bytes, each subsequent denomination being one thousand times the size of the previous: Byte; Kilobyte; Megabyte; Gigabyte; Terabyte; Petabyte; 1

4 Chapter 1: Introduction to big data Exabyte; Zettabyte; and Yottabyte. Mobile telephones can be used to illustrate the phenomenal growth of data in the past two decades. Today the average mobile phone purchaser will compare, amongst other features, the amount of storage space offered on devices in gigabytes. Going back just 20 years, there was no commercially available medium that would hold a single gigabyte. The fact that there is now the need for so much data capacity on a mobile phone device, and a much larger amount on any computer, displays the growth in data in such a short space of time. Current estimates suggest that there are between 8 10 billion devices connected (to the internet) in circulation. Many of those connected devices generate and post information to websites and data repositories with incredible frequency. The world revolves around data. Factors leading to big data It is important to understand a little of the background parameters that have enabled this scenario to come about. Most commentaries put the new world of big data down to a handful of factors which will be discussed in the following sections. Scientific research The ability to not only generate, but also to store and allow manipulation of data has given science (including medicine) a platform for research that has opened up many areas of study. One of the most reported cases of big data in science is that of the large hadron collider project, which uses around 150 million sensors. Each of these sensors can report at a rate of 40 million times per second. The reason for these numbers is that the collider can produce millions of collisions per second, and in order to extract any meaningful information, the data from these collisions needs to be collected and then processed. Of course not all collisions produce useful data, but in order to get to those ones that do, all the potential data needs to be collected and analysed. Any attempt to manage this information flow using conventional database tools is not likely to succeed. In some senses, science research represented 2

5 the first foray into big data. Before technological advances could provide the necessary computing power, research scientists were harnessing the unused processing power from a network of volunteers own equipment to provide data about projects that standard methods could not produce. Social media Sites such as Facebook and Twitter have grown enormously since their inception. These sites not only contain data about their user base data which could easily be dealt with using conventional data storage methods but also vast swathes of user generated data that is much more difficult to deal with. For example, as of October 2012, Facebook stored 220 billion photographs. 1 Considering that this is just one example of the many types of data Facebook stores, it is possible to see how this kind of enterprise generates volumes of material that were previously unheard of. Internet retailing Retailers produce enormous quantities of data through the combination of product information, customer information, financial transactions, and other peripheral data. Whilst this been the case for a while, the ability to sell products over the internet has given rise to a much larger quantum of information. Where previously a transaction may not have involved much, if any, customer details, a transaction over the internet will more than likely be linked to a customer account, and all the incumbent information held therein. Accumulation of these transactions from customers over time gives rise to much more data, and when combined with other customer transactions, provides the basis on which marketing departments, production specialists, and various other interested parties to analyse and use, and generate more data in the process. Given that there are very few retailers that do not sell over the internet (and certainly no one outside of the SME categorisation), the amounts of data generated are vast and ever increasing. Web search engines In May 2013, a Netcraft survey revealed that there were over 672 million active websites on the internet. 2 Browsers are often led to those websites by internet search engines such as Google. In order for Google to be able to 3

6 Chapter 1: Introduction to big data show websites that relate to search strings entered into its mechanism, it has to find, sort, and store information relating to those sites. Google, and similar internet search companies, also generate information from users in order to produce information that is valuable to organisations that want to sell goods and services, or direct browsers to their websites for other purposes. This all adds up to colossal quantities of data to be stored and utilised. Google alone employ somewhere near one million servers to put this information onto users desktops. What do these factors mean? The combination of the four factors above along with other mass data producing events ensures that the amount of data generated is always increasing, and thus the tools and techniques required to hold and manipulate it into useful information have to cope not only with today s volumes, but also the best guess at what tomorrows volume s will look like. Whilst the above factors are able to generate data, it would be largely transient were it not for the rapid improvements that have been engineered on the physical side of computing. The physical backbone for all this data has come about via a massively improved capacity to build fast, redundant storage repositories. Not only is storage more scalable and cheaper, it is also more intelligent, allowing systems to run important or frequently used data on expensive and faster drives, and less frequently used data on slower and cheaper drives. This technique is used for many transactional systems where users patience is likely to be stretched by delayed responses, for example in online banking, but at the same time it is relatively economical to build and maintain, which in turn is important for keeping banking transaction charges low enough to attract custom. Increase in internet bandwidth, and the availability of cheap connectivity, has given big data the connectivity needed in order to grow. Virtualisation of servers has also been a key component of this revolution, allowing maximum use of physical resources by multiple systems. This ability to run multiple systems on a single piece of machinery meant that end users of all sizes suddenly had access to facilities that which would have previously been beyond their economic means. Today a small business or individual can set up and run a transactional website without having to own 4

7 any physical hardware or software. It can also be achieved without much in the way of expertise, as there are so many free or inexpensive tools that allow users to build sites and repositories quickly and easily. All these factors combined created not only vast quantities of data, but also quantities that were increasing exponentially. Conventional tools were inadequate for dealing with this, so a transition to another type of system had to be brought about. This was largely achieved by a combination of work done by open source computing engineers working for internet search companies, and will be expanded upon later in this report. At this point it is important to stress that big data does not consist of the same sort of standards that conventional relational data systems adhere to. There are some standard tools in use, but the methods and practices of those companies manipulating the data are not the subject of a collaborative approach either within the industry or via the world of education. One could speculate that this could change given time. Also, given that technology changes have a habit of trickling down from early adopters to a wider audience, we some standards should evolve. The three V s Big data is often portrayed as having three principal characteristics of volume, velocity and variety the three V s: Volume: This is a fairly obvious characteristic given the nature of the title big data. The volumes of data being generated and processed are enormous. The example can be made of a goods warehouse attached to a retail chain. The warehouse may be able to cope with a thousand pallets a day coming in to the system and subsequently being selected for dispersal to retail outlets. As the volumes increase and decrease slightly, the system can flex to accommodate them. This is akin to a traditional relational data model. Should the volume increase drastically to a million pallets arriving, the system and warehouse would not be in a position to cope. In these circumstances, space would need to be found wherever possible, and more distributed categorisations would need to be applied to deal with the volume. This is akin to a big data environment. 5

8 Chapter 1: Introduction to big data Velocity: In the warehouse example the system would ordinarily cope with a thousand pallets a day of input and output. The extended input of a million pallets is likely to generate demand from customers, so that the number of transactions suddenly increases. In the real world of tangible goods, this would be extremely difficult to cope with. In the data world, this extreme throughput describes the velocity issue. It is not just about volume, but also the speed at which transactions happen, so systems also need to cope with this. Variety: The final characteristic is the variety of types of goods coming into the system. Initially the system would have been set up to cope with a fixed number of varieties of object, with the scope to vary this from time to time. A sizable increase in the variety of objects entering the system will not be dealt with effectively, or at all, so a change in varietal handling is required. In the world of data, expansive ranges of data types are now seen where once only various forms of text were expected. In big data terms, variety also refers to the notion of different structural characteristics of data: those being structured, unstructured, and semistructured. Big data systems have to be able to take all the potential types of data in, not only now but as far into the future as engineers can predict, in order to cope with the ever expanding data types that evolve. It also has to be able to cope with the interaction of structured and unstructured data, as many, if not all, large data systems will contain both. Who uses big data The earliest incarnations of big data were driven by the work undertaken by engineers from various web search companies, and from open source engineering groups. Arguably, Google had the need for a big data solution before most commercial organisations, bearing in mind the information they were collecting and then using about web browsing, and taking into account the exponential expansion of the number and size of websites over a few years. Big data repositories are based on open source software platforms, so once the base material for dealing with the problem was developed it became available to all users, in theory, from almost day one. Google adopted and adapted these techniques for their data stores, and 6

9 others followed. Nowadays all search engine data is likely to be based around big data techniques. Law firm clients from a varied range of industries nowadays use big data. Medicine Big data techniques are pertinent to medicine and healthcare, whether it relates to drugs companies research into products, healthcare professionals researching diseases and illness patterns, or healthcare bodies looking into predicting outbreaks of illnesses and producing best practice guidelines. The world of medicine has always produced large quantities of data, but it has not always been able to use it to best advantage. Scientific research The large hadron collider project has already been discussed as an obvious example of the scientific uses of big data. There are many scientific research projects that are capable of generating and using huge volumes of data, from weather forecasting to concrete sampling. For science, big data is a huge step forward. Retail Retail firms generate vast quantities of data through the enormous numbers of transactions that they produce. The biggest retailers can generate a million transactions per hour, and each transaction will have associated data about the purchaser, the product, and about the related purchases for example. Collating this data into usable silos allows retailers to analyse purchasers spending patterns in almost innumerable ways, giving them marketing material, and theoretically market advantage. Social media Sites such as Facebook, Twitter, and LinkedIn all generate vast quantities of data, predominantly through the Web 2.0 principal of user interaction. Sites of this nature allow users to post information, pictures and videos, share applications, and perform many other social tasks. All of this data is stored and re-used. The quantities involved are staggering, but the variety of types of data, and the speed at which data is uploaded, moved, edited, and deleted 7

10 Chapter 1: Introduction to big data is equally staggering. These websites are built around big data techniques because there is simply no realistic alternative. Law firms Whilst law as a profession is not an early adopter of big data techniques, there are clear indications that it is starting to use big data in several ways. Lawyers consume vast quantities of data in their day to day work. The additional techniques brought to the table via big data are going to enable them to produce additional results both on the client side and in the internal administrative areas. Big data uses for law firms will be discussed at length later in this report. Other users It is not difficult to see that insurance companies, pressure groups, manufacturers, and all manner of academics can all use big data techniques to make changes to the way they go about their business. The ability to harness so much additional data, process it into useful information and extract patterns that were previously missed is a tremendous opportunity that few if any are likely to pass up on. The opportunities that accompany big data techniques are truly unlimited. What big data is used for Big data techniques are currently in use in a variety of commercial, educational, and research institutions. The ability to combine structured data from incumbent systems with the mass of unstructured data that is being generated provides for an almost infinite amount of possibilities for its use. Whilst there is in theory no limit, only the larger and more technically adept are currently able to utilise it at present. There are many, wildly varying, predictions on the growth of big data. What can safely be said is that it will be adopted by more and more institutions, and the skillsets and tools that accompany it will become more prevalent. For now let s consider a few of the current uses of big data techniques. It almost goes without saying that big data techniques are used for storing the largest data repositories that are currently in use. The technology involved will be discussed later in this report. It is that technology, and particular its advantages over standard data storage techniques that allow such vast 8

11 accumulations of data. Of course data storage in itself is not a reason for doing anything. Data has to be usable, useful, timely, and accurate, so the technology that allows repositories to be built is also that which is used to extract from them. In many respects, marketing is the most obvious use of big data, given that it is the way in which the use of big data sets ends up in front of users on a day to day basis. Surfing the internet for most users will invariably include targeted adverts from suppliers. Often those adverts happen to be in one or more of the targeted persons spheres on interest. This is brought about by analysing a combination of the browsing patterns of internet surfers and their shopping patterns. With either set of data there is plenty of scope for targeted advertising, but with combinations of them the adverts tend to become more accurate, for want for a better expression. Google, Amazon, and other larger internet organisations use big data techniques to drive these adverts to consumers, and to create revenue in turn. The majority of Google s income is derived from advertising. Internet companies are not alone in their interest in using big data techniques for targeted marketing. Any organisation can now delve into this area, assuming they have access to the techniques involved, and to data sets. Many organisations will be generating their own data, but there is also the possibility of buying data in, and much of this will come from internet search material. The ability to harness not only large data sets, but also the incidental information that can be gleaned from them, can provide market advantage for retailers. The use of radio-frequency identification (RFID) using radio-frequency electromagnetic fields to transfer data for the purposes of automatically identifying and tracking tags has been widely adopted in retail. RFID tags contain an array of information about a product, from its base material to the location it is being vended in. Mixing this with other data, say weather patterns at the time of purchase, could give the retailer information on how sales are affected by weather, and thus combine weather forecasting into stock allocations to meet demand that fluctuates with weather patterns. Analysts also talk of using the data not just to analyse what they sell, but also what they may be missing in terms of sales (that is, why someone has not bought their product) thus enabling them to target those sales that would ordinarily have gone elsewhere. 9

12 Chapter 1: Introduction to big data Many law firm clients use and indeed rely on business intelligence (BI) tools to shape and administer their businesses. BI is concerned with recording and interpreting the activity of an organisation from any aspect in order to garner information about how the business performs from every angle. Big data provides new avenues for business intelligence to exploit, offering more data possibilities which, when combined with traditional or structured data, give analysts more scope, which in turn should deliver better and more up-to-date information. In a competitive world, big data is becoming a key success factor for those that can understand and/or use it effectively. This will increasingly include law firms either directly through their own big data use or indirectly through their clients big data activities. The possibilities are endless with big data and all organisations should be aware of its potential. Much is being made of the value of big data to the medical profession. The ability to collate and analyse masses of additional data, or to aggregate multiples of existing data will leverage huge increases of information to all areas of medical practice and research. Google currently have a high profile project mapping flu trends. This trend analysis comes from collecting data about users searching for flu information, which Google have demonstrated is closely linked to actual flu occurrences. Demonstrating where flu outbreaks may occur gives medical planners the knowledge they need to plan vaccinations and other preventative measures. In the financial world, fraud is often investigated after the fact. With big data techniques, institutions are starting to measure and assess more data and thus more patterns, which goes some way to predicting and therefore preventing certain types of fraud in real time. Big data also has an impact on risk assessment and management. Actuaries and insurance institutions are already getting a grip on big data techniques to refine their work. By including the wealth of peripheral data that these techniques bring, insurers and actuaries can assess not only more accurately, but more personally, so that eventually individuals will be measured on personal risk terms rather than in groups. The techniques are also leading to risks being identified earlier, or with more accuracy, so that risk management can be more accurately applied. For lawyers, this is an important step, as there are various forms of risk to assess, both with clients, and with case outcomes. 10

13 These are just a few examples of where big data is currently being used. There is clearly no definable upper limit as to what can be achieved with big data. Although the tools and techniques are currently limited to larger institutions, this will not be the case for too long, and we can expect to see widespread adoption in every marketplace. References 1. See: 2. Netcraft, May 2013 Web Server Survey, see: archives/2013/04/19/may-2013-web-server-survey.html. 11