Big Data Deep Learning: Challenges and Perspectives

Transcription

1 Big Data Deep Learning: Challenges and Perspectives D.saraswathy Department of computer science and engineering IFET college of engineering Villupuram Abstract Deep learning is currently an extremely active research area in machine learning and pattern recognition society. It has gained huge successes in a broad area of application such as speech recognition, computer vision, and natural language processing. With the sheer size of data available today, big data brings big opportunities and transformative potential for various sectors. On the other hand, it also presents unprecedented challenges to harnessing data and information. As the data keeps getting bigger, deep learning is coming to play a key role in providing big data predictive analytics solution. In this paper, provide a brief overview of deep learning, and highlight big data, as well as the future trends. Keywords Classifier design and evaluation, feature representation, machine learning, neural nets models, parallel processing. I. INTRODUCTION Deep learning and Big Data are two hottest trends in the rapidly growing digital world. While Big Data has been defined in different ways, herein it is referred to the exponential growth and wide availability of digital data that are difficult or even impossible to be managed and analyzed using conventional software tools and technologies. Digital data, in all shapes and sizes, is growing at astonishing rates. For example, according to the National Security Agency, the Internet is processing 1,826 Petabytes of data per day [1]. In 2011, digital information has grown nine times in volume in just five years [2] and by 2020, its amount in the world will reach 35 trillion gigabytes [3]. This explosion of digital data brings big opportunities and transformative potential for various sectors such as enterprises, healthcare industry manufacturing, and educational services [4]. It also leads to a dramatic paradigm shift in our scientific research towards data-driven discovery. While Big Data offers the great potential for revolutionizing all aspects of our society, harvesting of valuable knowledge from Big Data is not an ordinary task. The large and rapidly growing body of information hidden in the unprecedented volumes of non-traditional data requires both the development of advanced technologies and interdisciplinary teams working in close collaboration. Today, machine learning techniques, together with advances in available computational power, have come to play a vital role in Big Data analytics and knowledge discovery (see[5]-[8]). They are employed widely to leverage the Predictive power of Big Data in fields like search engines, medicine, and astronomy. As an extremely active subfield of machine learning, deep learning is considered, together with Big Data, as the big deals and the bases for an American innovation and economic revolution [9]. In contrast to most conventional learning methods, which are considered using shallow-structured learning architectures, deep learning refers to machine learning techniques that use supervised and/or unsupervised strategies to automatically learn hierarchical representations in deep architectures for classification [10], [11]. Inspired by biological observations on human brain mechanisms for processing of natural signals, deep learning has attracted much attention from the academic community in recent years due to its state-of-the-art performance in many research domains such as speech recognition [12], [13], collaborative fultering [14], and computer vision [15], [16]. Deep learning has also been successfully applied in industry products that take advantage of the large volume of digital data. Companies like Google, Apple, and Facebook, who collect and analyze massive amounts of data on a daily basis, have been aggressively pushing forward deep learning related projects. For example, Apple s Siri, the virtual personal assistant in iphones, offers a wide variety of services including weather reports, sport news, answers to user s questions, and reminders etc. by utilizing deep learning and more and more data collected by Apple services [17]. Google applies deep learning algorithms to massive chunks of messy data obtained from the Internet for Google s translator Android s voice recognition, Google s street view, and image search engine [18]. Other industry giants are not far behind either. For example, Microsoft s real-time language translation in Bing voice search [19] and IBM s brain-like computer [18], [20] use techniques like deep learning to leverage Big Data for competitive advantage. As the data keeps getting bigger, deep learning is coming to play a key role in providing big data predictive analytics solutions, particularly with the increased processing power and the advances in graphics processors. In this paper, our goal is not to present a comprehensive survey of all the related work in deep learning, but mainly to discuss the most important issues related to learning from massive amounts of data, highlight current research efforts and the challenges to big data, as well as the future trends. The rest of the paper is organized as follows. Section 2 presents a brief review of two commonly used deep learning

2 architectures. Section 3 discusses the strategies of deep learning from massive amounts of data. Finally, we discuss the challenges and perspectives of deep learning for Big Data in Section 4. RBMs is a parameter predetermined by users and pre-training requires only unlabeled data (for good generalization). II. OVERVIEW OF DEEP LEARNING Deep learning refers to a set of machine learning techniques that learn multiple levels of representations in deep architectures. In this section, we will present a brief overview of two well-established deep architectures: deep belief net-works (DBNs) [21]-[23] and convolutional neural networks (CNNs) [24]-[26]. A. Deep belief network Conventional neural networks are prone to get trapped in local optima of a non-convex objective function, which often leads to poor performance [27]. Furthermore, they cannot take advantage of unlabeled data, which are often abundant and cheap to collect in Big Data. To alleviate these problems, a deep belief network (DBN) uses a deep architecture that is capable of learning feature representations from both the labeled and unlabeled data presented to it [21]. It incorporates both unsupervised pre-training and supervised fine-tuning strategies to construct the models: unsupervised stages intend to learn data distributions without using label information and supervised stages perform local search for fine tuning. Fig. 1 shows a typical DBN architecture, which is composed of a stack of Restricted Boltzmann Machines (RBMs) and/or one or more additional layers for discrimination tasks. RBMs are probabilistic generative models that learn a joint probability distribution of observed (training) data without using data labels [28]. They can effectively utilize large amounts of unlabeled data for exploiting complex data structures. Once the structure of a DBN is determined, the goal for training is to learn the weights (and biases) between layers. This is conducted firstly by an unsupervised learning of RBMs. A typical RBM consists of two layers: nodes in one layer are fully connected to nodes in the other layer and there is no connection for nodes in the same layer (see Fig.1, for example, the input layer and the first hidden layer H 1 form a RBM) [28]. Consequently, each node is independent of other nodes in the same layer given all nodes in the other layer. This characteristic allows us to train the generative weights W of each RBMs using Gibbs sampling [29], [30]. Before fine-tuning, a layer-by-layer pre-training of RBMs is performed: the outputs of a RBM are fed as inputs to the next RBM and the process repeats until all the RBMs are pre-trained. This layer-by-layer unsupervised learning is critical in DBN training as practically it helps avoid local optimal and alleviates the over-fitting problem that is observed when millions of parameters are used. Furthermore, the algorithm is very efficient in terms of its time complexity, which is linear to the number and size of RBMs [21].Features at different layers contain different information about data structures with higher-level features constructed from lower-level features. Note that the number of stacked Fig 1. Illustration of a deep belief network architecture. As a generative mode, the RBM training includes a Gibbs sampler to sample hidden units based on the visible units and vice versa (Eqs. (1) and (2)). The weights between these two layers are then updated using the CD rule (Eq. 3). This process will repeat until convergence. An RBM models data distribution using hidden units without employing label information. This is a very useful feature in Big Data analysis as DBN can potentially leverage much more data (without knowing their labels) for improved performance. After pre-training, information about the input data is stored in the weights between every adjacent layers. The DBN then adds a final layer representing the desired outputs and the overall network is fine-tuned using labeled data and back propagation strategies for better discrimination (in some implementations, on top of the stacked RBMs, there is another layer called associative memory determined by supervised learning methods). There are other variations for pre-training: instead of using RBMs, for example, stacked denoising auto-encoders [32], [33] and stacked predictive sparse coding [34] are also proposed for unsupervised feature learning. Furthermore, recent results show that when a large number of training data is available, a fully supervised training using random initial weights instead of the pre-trained weights (i.e., without using RBMs or auto-encoders) will practically work well [13], [35]. For example, a discriminative model starts with a network with one single hidden layer (i.e., a shallow neural network), which is trained by back propagation method. Upon convergence, a new hidden layer is inserted into this shallow NN (between the first hidden layer and the desired output layer) and the full network is discriminatively trained again. This process is continued until a predetermined criterion is met (e.g., the number of hidden neurons).

3 B. Convolutional Neural Networks A typical CNN is composed of many layers of hierarchy with some layers for feature representations (or feature maps) and others as a type of conventional neural networks for classification [24]. It often starts with two altering types of layers called convolutional and subsampling layers: convolutional layers perform convolution operations with several filter maps of equal size, while subsampling layers reduce the sizes of proceeding layers by averaging pixels within a small neighborhood (or by maxpooling [36], [37]). Fig. 2 shows a typical architecture of CNNs. The input is first convoluted with a set of filters (C layers in Fig. 2). These 2D filtered data are called feature maps. After a nonlinear transformation, a subsampling is further performed to reduce the dimensionality (S layers in Fig. 2). The sequence of convolution/subsampling can be repeated many times (pre-determined by users). In summary, inspired by biological processes [40], CNN algorithms learn a hierarchical feature representation by utilizing strategies like local receptive fields (the size of each filter is normally small), shared weights (using the same weights to construct all the feature maps at the same level significantly reduces the number of parameters), and subsampling (to further reduce the dimensionality). Each filter bank can be trained with either supervised or unsupervised methods. A CNN is capable of learning good feature hierarchies automatically and providing some degree of translational and distortional invariances. III. Deep Learning For Massive Amounts Of Data While deep learning has shown impressive results in many applications, its training is not a trivial task for Big Data learning due to the fact that iterative computations inherent in most deep learning algorithms are often extremely difficult to be parallelized. Thus, with the unprecedented growth of commercial and academic data sets in recent years, there is a surge in interest in effective and scalable parallel algorithms for training deep models [12], [13], [15], [41]-[44]. In contrast to shallow architectures where few parameters are preferable to avoid over fitting problems, deep learning algorithms enjoy their success with a large number of hid-den neurons, often resulting in millions of free parameters. Thus, large-scale deep learning often involves both large volumes of data and large models. Some algorithmic approaches have been explored for large-scale learning: for example, locally connected networks [24], [39], improved optimizers [42], and new structures that can be implemented inparallel [44]. Recently, Deng et al. [44] proposed a modified deep architecture called Deep Stacking Network (DSN), which can be effectively parallelized. A DSN consists of several specialized neural networks (called modules) with a single hidden layer. Stacked modules with inputs composed of raw data vector and the out puts from previous module form a DSN. Most recently, a new deep architecture called Tensor Deep Stacking Network (T-DSN), which is based on the DSN, is implemented using CPU clusters for scalable parallel computing [45]. The use of great computing power to speed up the training process has shown significant potential in Big Data deep learning. For example, one way to scale up DBNs is to use multiple CPU cores, with each core dealing with a subset of training data (data-parallel schemes). Vanhouckeet al. [46] discussed some aspects of technical details, including carefully designing data layout, batching of the computation, using SSE2 instructions, and leveraging SSE3 and SSE4 instructions for fixedpoint implementation. These implementations can enhance the performance of modern CPUs more for deep learning. Another recent work aims to parallelize Gibbs sampling of hidden and visible units by splitting hidden units and visible units into n machines, each responsible for 1/n of the units [47]. In order to make it work, data transfer between machines is required (i.e., when sampling the hidden units, each machine will have the data for all the visible units and vice versa). This method is efficient if both the hidden and visible units are binary and also if the sample size is modest. The communication cost, however, can rise up quickly if large-scale data sets are used. Other methods for large-scale deep learning also explore FPGAbased implementation [48] with a custom architecture: a control unit implemented in a CPU, a grid of multiple full-custom processing tiles, and a fast memory. In this survey, we will focus on some recently developed deep learning frameworks that take advantage of great computing power available today. Take Graphics Process (GPUs) as an example: as of August 2013, NVIDIA single precision GPUs exceeded 4.5 TeraFLOP/s with a memory bandwidth of near 300 GB/s [49]. They are particularly suited for massively parallel computing with more transistors devoted for data proceeding needs. These newly developed deep learning frameworks have shown significant advances in making large-scale deep learning practical.

4 Fig 3. An illustrative architecture of a CUDA-capable GPU with highly threaded streaming processors (SPs) Fig. 3 shows a schematic for a typical CUDA- capable GPU with four multi-processors. Each multi-processor (MP) consists of several streaming multiprocessors (SMs) to form a building block (Fig. 3 shows two SMs for each block). Each SM has multiple stream processors (SPs) that share control logic and low-latency memory. Furthermore, each GPU has a global memory with very high bandwidth and high latency when accessed by the CPU (host). This architecture allows for two levels of parallelism: instruction (memory) level (i.e.,mps) and thread level (SPs). This SIMT (Single Instruction,Multiple Threads) architecture allows for thousands or tens of thousands of threads to be run concurrently, which is best suited for operations with large number of arithmetic operations and small access times to memory. Such levels of parallelism can also be effectively utilized with special attention on the data flow when developing GPU parallel computing applications. One consideration, for example, is to reduce the data transfer between RAM and the GPU s global memory [50] by transferring data with large chunks. This is achieved by uploading as large sets of unlabeled data as possible and by storing free parameters as well as intermediate computations, all in global memory. In addition, data parallelism and learning updates can be implemented by leveraging the two levels of parallelism: input examples can be assigned across MPs, while individual nodes can be treated in each thread (i.e., SPs). A. Large-Scale Deep Belief Networks Raina et al. [41] proposed a GPU-based framework for massively parallelizing unsupervised learning models including DBNs (in this paper, they refer the algorithms to stacked RBMs) and sparse coding [21]. While previous models tend to use one to four million free parameters (e.g., Hinton & Salakhutdinov [21] used 3.8 million parameters for free images and Ranzato and Szummer used three million parameters for text processing [51]), the proposed approach can train on more than 100 million free parameters with millions of unlabeled training data [41]. Because transferring data between host and GPU global memory is time consuming, one needs to minimize hostdevice transfers and take advantage of shared memory. To achieve this, one strategy is to store all parameters and a large chunk of training examples in global memory during training [41]. This will reduce the data transfer times between host and globa memory and also allow for parameter updates to be carried out fully inside GPUs. In addition, to utilize the MP/SP levels of parallelism, a few of the unlabeled training data in global memory will be selected each time to compute the updates concurrently across blocks (data parallelism). B. Large-Scale Convolutional Neural Networks CNN is a type of locally connected deep learning methods. Large-scale CNN learning is often implemented on GPUs with several hundred parallel processing cores. CNN training involves both forward and backward propagation. For parallelizing forward propagation, one or more blocks are assigned for each feature map depending on the size of maps [36]. Each thread in a block is devoted to a single neuron in a map. Consequently, the computation of each neuron, which includes convolution of shared weights (kernels) with neurons from the previous layers, activation, and summation, is performed in a SP. The outputs are then stored in the global memory. C. Combination Of Data- And Model-Parallel Schemes DistBelief is a software framework recently designed for dis-tributed training and learning in deep networks with very large models (e.g., a few billion parameters) and largescale data sets. It leverages large-scale clusters of machines to manage both data and model parallelism via multithreading, message passing, synchronization as well as communication between machines [56]. For large-scale data with high dimensionality, deep learning often involves many densely connected layers with a large number of free parameters (i.e., large models). To deal with large model learning, DistBelief first implements model parallelism by allowing users to partition large network architectures into several smaller structures (called blocks), whose nodes will be assigned to and calculated in several machines (collectively we call it a partitioned model ). Each block will be assigned to one machine. Boundary nodes (nodes whose edges belong to more than one partition) require data transfer between machines. Apparently, fully-connected networks have more boundary nodes and often demand higher communication costs than locally-connected structures, and thus less performance benefits. Nevertheless, as many as 144 partitions have been reported for large models in, which leads to significant improvement of training speed. DistBelief also implements data parallelism and employs two separate distributed optimization procedures: Downpour stochastic gradient descent (SGD) and Sandblaster [56], which perform online and batch optimization, respectively. Herein we will discuss Downpour in details and more information about Sandblaster can be found in the reference

5 [56]. First, multiple replicas of the partitioned model will be created for training and inference. Like deep learning models, large data sets will be partitioned into many subsets. DistBelief will then run multiple replicas of the partitioned model to compute gradient descent via Downpour SGD on different subsets of training data. Specifically, DistBelief employs a centralized parameter server storing and applying updates for all parameters of the models. Parameters are grouped into server shards. At any given time, each machine in a partitioned model needs only to communicate with the parameter server shards that hold the relevant parameters. This communication is asynchronous: each machine in a partitioned model runs independently and each parameter server shard acts independently as well. One advantage of using asynchronous communication over standard synchronous SGD is its fault tolerance: in the event of the failures of one machine in a model copy, other model replicas will continue communicating with the central parameter server to process the data and update the shared weights. deep learning architecture consists of three stacked layers, each with sublayers of local filtering, local pooling, and local contrast normalization. The filtering sublayers are not convolutional, each filter with its own weights. The optimization of this architecture involves an overall objective function that is the summation of the objective functions for the three layers, each aiming at minimizing a reconstruction error while maintaining sparsity of connections between sublayers. The DistBelief framework is able to scale up the dataset, the model, and the resources all together. The model is partitioned into 169 machines, each with 16 CPU cores. Multiple cores allow for another level of parallelism where each subset of cores can perform different tasks. Asynchronous SGD is implemented with several replicas of the core model and mini-batch of training examples. The framework was able to train as many as 14 million images with a size of 200 by 200 pixels and more than 20 thousand categories for three days over a cluster of 1,000 machines with 16,000 cores The model is capable of learning high-level features to detect objects without using labeled data. IV. REMAINING CHALLENGES AND PERSPECTIVES: DEEP LEARNING FOR BIG DATA Fig 5. DistBelief: models are partitioned into four blocks and consequently assigned to four machines. In practice, the Adagrad adaptive learning rate procedure [57] is integrated into the Downpour SGD for better performance. DistBelief is implemented in two deep learning models: a fully connected network with 42 million model parameters and 1.1 billion examples, and a locallyconnected convolutional neural network with 16 million images of 100 by 100 pixels and 21,000 categories (as many as 1.7 billion parameters). The experimental results show that locally connected learning models will benefit more from DistBelief: indeed, with 81 machines and 1.7 billion parameters, the method is 12x faster than using a single machine. As demonstrated in [56], a significant advantage of DistBelief is its ability to scale up from single machine to thousands of machines, which is the key to Big Data analysis. Most recently, the DistBelief framework was used to train a deep architecture with a sparse deep autoencoder, local receptive fields, pooling, and local contrast normalization. The In recent years, Big Data has taken center stage in government and society at large. In 2012, the Obama Administration announced a Big Data Research and Development Initiative to help solve some of the Nation s most pressing challenges [60]. Consequently, six Federal departments and agencies (NSF, HHS/NIH, DOD, DOE, DARPA, and USGS) committed more than $200 million to support projects that can transform our ability to harness in novel ways from huge volumes of digital data. In May of the same year, the state of Massachusetts announced the Massachusetts Big Data Initiative that funds a variety of research institutions [61]. In April, 2013, U.S. President Barack Obama announced another federal project, a new brain mapping initiative called the BRAIN (Brain Research Through Advancing Innovative Neurotechnologies) [62] aiming to develop new tools to help map human brain functions, understand the complex links between function and behavior, and treat and cure brain disorders. This initiative might test and extend the current limits of technologies for Big Data collection and analysis, as NIH director Francis Collins stated that collection, storage, and processing of yottabytes (a billion petabytes) of data would eventually be required for this initiative. While the potential of Big Data is undoubtedly significant, fully achieving this potential requires new ways of thinking and novel algorithms to address many technical challenges. For example, most traditional machine learning algorithms were designed for data that would be completely loaded into memory. With the arrival of Big Data age, however, this assumption does not hold any more. Therefore, algorithms that can learn from massive amounts of data are needed.in spite of all the recent achievement in large-scale deep learning as discussed in Section 3, this field is still in its infancy. Much more needs to be done to address

6 many significant challenges posted by Big Data, often characterized by the three V s model: volume, variety, and velocity [63], which refers to large scale of data, different types of data, and the speed of streaming data, respectively. A. Deep Learning From High Volumes Of Data First and foremost, high volumes of data present a great challenging issue for deep learning. Big data often possesses a large number of examples (inputs), large varieties of class types (outputs), and very high dimensionality (attributes). These properties directly lead to running-time complexity and model complexity. The sheer volume of data makes it often impossible to train a deep learning algorithm with a central processor and storage. Instead, distributed frameworks with parallelized machines are preferred. Recently, impressive progresses have been made to mitigate the challenges related to high volumes. The novel models utilize clusters of CPUs or GPUs in increasing the training speed without scarifying accuracy of deep learning algorithms. Strategies for data parallelism or model parallelism or both have been developed. For example, data and models are divided into blocks that fit with in-memory data; the forward and backward propagations can be implemented effectively in parallel [56], [58], although deep learning algorithms are not trivially parallel. The most recent deep learning framework can handle a significantly large number of samples and parameters. It is also possible to scale up with more GPUs used. It is less clear, however, how the deep learning systems can continue scaling significantly beyond the current framework. While we can expect the continuous growth in computer memory and computational power (mainly through parallel or distributed computing environment), further research and effort on addressing issues associated with computation and communication management (e.g., copying data or parameters or gradient values to different machines) are needed for scaling-up to very large data sets. Ultimately, to build the future deep learning system scalable to Big Data, one needs to develop high performance computing infrastructure-based systems together with theoretically sound parallel learning algorithms or novel architectures. Another challenge associated with high volumes is the data incompleteness and noisy labels. Unlike most conventional datasets used for machine learning, which were highly curated and noise free, Big Data is often incomplete resulting from their disparate origins. To make things even more complicated, majority of data may not be labeled, or if labeled, there exist noisy labels. Take the 80 million tiny image database as an example, which has 80 million low-resolution color images over 79,000 search terms [64]. This image database was created by searching the Web with every non-abstract English noun in the WordNet. Several search engines such as Google and Flickr were used to collect the data over the span of six months. Some manual curation was conducted to remove duplicates and low-quality images. Still, the image labels are extremely unreliable because of search technologies. One of the unique characteristics deep learning algorithms possess is their ability to utility unlabeled data during train-ing: learning data distribution without using label information. Thus, the availability of large unlabeled data presents ample opportunities for deep learning methods. While data incompleteness and noisy labels are part of the Big Data package, we believe that using vastly more data is preferable to using smaller number of exact, clean, and carefully curated data. Advanced deep learning methods are required to deal with noisy data and to be able to tolerate some messiness. For example, a more efficient cost function and novel training strategy may be needed to alleviate the effect of noisy labels. Strategies used in semisupervised learning [65]-[68] may also help alleviate problems related to noisy labels. B. Deep Learning For High Variety Of Data The second dimension for Big Data is its variety, i.e., data today comes in all types of formats from a variety sources, probably with different distributions. For example, the rapidly growing multimedia data coming from the Web and mobile devices include a huge collection of still images, video and audio streams, graphics and animations, and unstructured text, each with different characteristics. A key to deal with high variety is data integration. Clearly, one unique advantage of deep learning is its ability for representation learning - with either supervised or unsupervised methods or combination of both, deep learning can be used to learn good feature representations for classification. It is able to discover intermediate or abstract representations, which is carried out using unsupervised learning in a hierarchy fashion: one level at a time and higher-level features defined by lower-level features. Thus, a natural solution to address the data integration problem is to learn data representations from each individual data sources using deep learning methods, and then to integrate the learned features at different levels. Deep learning has been shown to be very effective in integrating data from different sources. For example, Ngiamet al. [69] developed a novel application of deep learning algorithms to learn representations by integrating audio and video data. They demonstrated that deep learning is generally effective in (1) learning single modality representations through multiple modalities with unlabeled data and (2) learning shared representations capable of capturing cor-relations across multiple modalities. Most recently, Srivastava and Salakhutdinov [70] developed a multimodal Deep Boltzmann Machine (DBM) that fuses two very different data modalities, real-valued dense image data and text data with sparse word frequencies, together to learn a unified representation. DBM is a generative model without finetuning: it first builds multiple stacked-rbms for each modality; to form a multimodal DBM, an additional layer of binary hidden units is added on top of these RBMs for joint representation. It learns a joint distribution in the multimodal input space, which allows for learning even with missing modalities. While current experiments have demonstrated that deep learning is able to utilize

7 heterogeneous sources for significant gains in system performance, numerous questions remain open. For example, given that different sources may offer conflicting information, how can we resolve the conflicts and fuse the data from different sources effectively and efficiently. While current deep learning methods are mainly tested upon bi-modalities (i.e., data from two sources), will the system performance benefits from significantly enlarged modalities? Furthermore, at what levels in deep learning architectures are appropriate for feature fusion with hetero-generous data? Deep learning seems well suited to the integration of heterogeneous data with multiple modalities due to its capability of learning abstract representations and the underlying factors of data variation. REFERENCES [1]. National Security Agency. The National Security Agency: Missions, Authorities, Oversight and Partnerships [Online]. [2] J. Gantz and D. Reinsel, Extracting Value from Chaos. Hopkinton, MA, USA: EMC, Jun [3] J. Gantz and D. Reinsel, The Digital Universe Decade Are You Ready. Hopkinton, MA, USA: EMC, May [4] (2011, May). Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute [Online]. Available: next_frontier_for_innovation [5] J. Lin and A. Kolcz, Large-scale machine learning at twitter, in Proc. ACM SIGMOD, Scottsdale, Arizona, USA, 2012, pp