Deep Learning For Text Processing

Size: px

Start display at page:

Download "Deep Learning For Text Processing"

Alyson Park
8 years ago
Views:

2 Deep Learning For Text Processing Jeffrey A. Bilmes Professor Departments of Electrical Engineering & Computer Science and Engineering University of Washington, Seattle Monday, July 14th, 2014 J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 2 / 6

Engineering University of Washington, Seattle http://melodi.ee.washington.

3 Deep Neural Networks Aspects: Deep Generative Models and Deep Non-linear Mappings (classification & regression) Convolutional Deep Models and Aggregation methods (sum, max pooling, etc.) Non-Linear Learning of Feature Representations, Autoencoders Distributed Representations (efficiency of parameter utilization). Applications and Successes: Computer Vision (image, video, motion capture) Speech (acoustic modeling in speech, speech perception, continuous representations of words/phrases, etc.) and NLP (language modeling, sentiment classification, etc.) Information Retrieval, learning similarity, deep CCA, etc. J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 3 / 6

Applications and Successes: Computer Vision (image, video, motion capture) Speech (acoustic modeling in speech, speech perception, continuous representations of words/phrases,

4 Deep Neural Networks Aspects: Deep Generative Models and Deep Non-linear Mappings (classification & regression) Convolutional Deep Models and Aggregation methods (sum, max pooling, etc.) Non-Linear Learning of Feature Representations, Autoencoders Distributed Representations (efficiency of parameter utilization). Applications and Successes: Computer Vision (image, video, motion capture) Speech (acoustic modeling in speech, speech perception, continuous representations of words/phrases, etc.) and NLP (language modeling, sentiment classification, etc.) Information Retrieval, learning similarity, deep CCA, etc. Resources: Big data - big data is different data, big neural networks are different neural networks Big computation - CPU only computation is insufficient, while GPUs are feasible J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 3 / 6

5 GPU vs. CPU Performance Over Time Source: GTC (GPU Tech. Conference) Keynote, NVIDIA CEO Jen-Hsun Huang, 3/25/2014 J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 4 / 6

6 Deep Models and GPUs J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 5 / 6

7 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

8 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. GPUs: commodity high-performance supercomputing (thanks to gaming industry). Hence, inexpensive and ubiquitous. J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

GPUs: commodity high-performance supercomputing (thanks to gaming industry).

9 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. GPUs: commodity high-performance supercomputing (thanks to gaming industry). Hence, inexpensive and ubiquitous. Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational needs are well-matched with what GPUs offer. J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational

10 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. GPUs: commodity high-performance supercomputing (thanks to gaming industry). Hence, inexpensive and ubiquitous. Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational needs are well-matched with what GPUs offer. Q: Could other machine learning methods (e.g., kernel machines, random forests, graphical models, etc.) have a correspondingly well matched, inexpensive, ubiquitous, and high performance hardware platform to run on? J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational needs are well-matched with what GPUs offer.

11 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. GPUs: commodity high-performance supercomputing (thanks to gaming industry). Hence, inexpensive and ubiquitous. Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational needs are well-matched with what GPUs offer. Q: Could other machine learning methods (e.g., kernel machines, random forests, graphical models, etc.) have a correspondingly well matched, inexpensive, ubiquitous, and high performance hardware platform to run on? Q: Or is the nature of such neural computations (matrix operations) such that it will always have an advantage due its regularity and hardware optimizability? J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

Q: Could other machine learning methods (e.g., kernel machines, random forests, graphical models, etc.

12 Questions Deep Learning needs big data to train, and is well-matched to GPUs (for matrix operations) to do so. GPUs: commodity high-performance supercomputing (thanks to gaming industry). Hence, inexpensive and ubiquitous. Part of the success of deep models is due to wave of GPU performance increase, and that deep models computational needs are well-matched with what GPUs offer. Q: Could other machine learning methods (e.g., kernel machines, random forests, graphical models, etc.) have a correspondingly well matched, inexpensive, ubiquitous, and high performance hardware platform to run on? Q: Or is the nature of such neural computations (matrix operations) such that it will always have an advantage due its regularity and hardware optimizability? Q: If so, can text/nlp applications like parsing, SMT be made just as regular? J. Bilmes Deep Learning For Text Processing Microsoft Research Faculty Summit, 2014 page 6 / 6

Sense Making in an IOT World: Sensor Data Analysis with Deep Learning

Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Natalia Vassilieva, PhD Senior Research Manager GTC 2016 Deep learning proof points as of today Vision Speech Text Other Search & information