PhD in Computer Science and Engineering Bologna, April Machine Learning. Marco Lippi. Marco Lippi Machine Learning 1 / 80

Transcription

1 PhD in Computer Science and Engineering Bologna, April 2016 Machine Learning Marco Lippi Marco Lippi Machine Learning 1 / 80

2 Recurrent Neural Networks Marco Lippi Machine Learning 2 / 80

3 Material and references Many figures are taken from Cristopher Olah s tutorial (colah.github.io) and Alex Karpathy s post The unreasonable effectiveness of Recurrent Neural Networks Marco Lippi Machine Learning 3 / 80

4 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input Marco Lippi Machine Learning 4 / 80

5 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input 1. Standard tasks such as image classification Marco Lippi Machine Learning 5 / 80

6 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input 2. Fixed input, sequence output: e.g., image captioning Marco Lippi Machine Learning 6 / 80

7 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input 3. Sequence classification: e.g., sentiment classification Marco Lippi Machine Learning 7 / 80

8 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input 4. Sequence input/output: e.g., machine translation Marco Lippi Machine Learning 8 / 80

9 Processing sequences In many tasks we should deal with variable-size input (text, audio, video) while most algorithms can deal with just fixed-size input 5. Synced sequence input/output: e.g., video frame classification Marco Lippi Machine Learning 9 / 80

10 Processing sequences One possibility: use convolutional neural networks with max pooling over all feature maps [Figure by Kim, 2014] Problem: transalation invariance... Marco Lippi Machine Learning 10 / 80

11 Processing sequences In a recurrent neural network the hidden state at time t depends on the input on the hidden state at time t 1 (memory) y t = f V (h t ) h t = f W (h t 1, x t ) Marco Lippi Machine Learning 11 / 80

12 Processing sequences We can parametrize: input-hidden connections hidden-hidden connections hidden-output connections y t = W hy h t h t = tanh(w hh h t 1 + W xh x t ) Marco Lippi Machine Learning 12 / 80

13 Recurrent Neural Network Language Model x(t) = w(t) + s(t 1) s j (t) = f ( i x i (t)u ji ) y k (t) = g( j s j (t)v kj ) [Figure by Mikolov et al., 2010] Wrt NNLM, no need to specify context dimension in advance! Marco Lippi Machine Learning 13 / 80

14 Recurrent Neural Networks (RNNs) A classic RNN can be unrolled through time, so that the looping connections are made explicit Marco Lippi Machine Learning 14 / 80

15 Recurrent Neural Networks (RNNs) Backpropagation Through Time (BPTT) [Figure from Wikipedia] Marco Lippi Machine Learning 15 / 80

16 Recurrent Neural Networks (RNNs) BPTT drawbacks: decide the value k for unfolding exploding or vanishing gradients exploding could be controlled with gradient clipping vanishing has to be faced with different models (LSTM) Marco Lippi Machine Learning 16 / 80

17 Recurrent Neural Networks (RNNs) Marco Lippi Machine Learning 17 / 80

18 Recurrent Neural Networks (RNNs) Some short- or mid-term dependencies can be afforded... Marco Lippi Machine Learning 18 / 80

19 Recurrent Neural Networks (RNNs) But the model fails in learning long-term dependencies! The main problem is vanishing gradients in BPTT... Marco Lippi Machine Learning 19 / 80

20 Long Short-Term Memory Networks (LSTMs) An LSTM is basically an RNN with a different computational block LSTMs were designed by Hochreiter & Schmidhuber in 1997! Marco Lippi Machine Learning 20 / 80

21 Long Short-Term Memory Networks (LSTMs) An LSTM block features a memory cell which can maintain its state over time, and non-linear gating units which regulate the information flow into and out of the cell [Greff et al., 2015] Marco Lippi Machine Learning 21 / 80

22 Long Short-Term Memory Networks (LSTMs) Cell state Just let the information go through the network Other gates can optionally let information through Marco Lippi Machine Learning 22 / 80

23 Long Short-Term Memory Networks (LSTMs) Forget gate Sigmoid layer that produces weights for the state cell C t 1 Decides what to keep (1) or forget (0) of past cell state f t = σ(w f [h t 1, x t ] + b f ) Marco Lippi Machine Learning 23 / 80

24 Long Short-Term Memory Networks (LSTMs) Input gate Allows novel information be used to update state cell C t 1 i t = σ(w i [h t 1, x t ] + b i ) C t = tanh(w C [h t 1, x t ] + b C ) Marco Lippi Machine Learning 24 / 80

25 Long Short-Term Memory Networks (LSTMs) Cell state update Combine old state (after forgetting) with novel input C t = f t C t 1 + i t C t Marco Lippi Machine Learning 25 / 80

26 Long Short-Term Memory Networks (LSTMs) Output gate Build the output to be sent to next layer and upper block o t = σ(w o [h t 1, x t ] + b O ) h t = o t tanh(c t ) Marco Lippi Machine Learning 26 / 80

27 Long Short-Term Memory Networks (LSTMs) Putting all together f t = σ(w f [h t 1, x t ] + b f ) i t = σ(w i [h t 1, x t ] + b i ) o t = σ(w o [h t 1, x t ] + b O ) h t = o t tanh(c t ) C t = f t C t 1 + i t C t Marco Lippi Machine Learning 27 / 80

28 Application: text generation Example: character-level language model [Figure by A. Karpathy] Marco Lippi Machine Learning 28 / 80

29 Application: text generation Example: character-level language model Pick up a (large) plain text file Feed the LSTM with the text Predict next character given past history At prediction time, sample from output distribution Get LSTM-generated text Marco Lippi Machine Learning 29 / 80

30 [Figure by A. Karpathy] Marco Lippi Machine Learning 30 / 80

31 Application: text generation Interpretation of cells (blue=off, red=on) [Figure by A. Karpathy] Marco Lippi Machine Learning 31 / 80

32 Application: text generation Interpretation of cells (blue=off, red=on) [Figure by A. Karpathy] Marco Lippi Machine Learning 32 / 80

33 Application: text generation Example: training with Shakespeare poems [Figure by A. Karpathy] Marco Lippi Machine Learning 33 / 80

34 Application: text generation Example: training with LaTeX math papers [Figure by A. Karpathy] Marco Lippi Machine Learning 34 / 80

35 Application: text generation Example: training with LaTeX math papers [Figure by A. Karpathy] Marco Lippi Machine Learning 35 / 80

36 Application: text generation Example: training with Linux Kernel (C code) [Figure by A. Karpathy] Marco Lippi Machine Learning 36 / 80

37 Application: image captioning [Figure by A. Karpathy] Marco Lippi Machine Learning 37 / 80

38 Application: handwriting text generation [Figure by Graves, 2014] Marco Lippi Machine Learning 38 / 80

39 Variants of the LSTM model Many different LSTM variants have been proposed: Peep-hole connections (almost standard now) all gates can also look at the state Coupling forget and input gates basically we input only when we forget Gated Recurrent Units (GRUs) where a single gate controls forgetting and update See LSTM: a search space odyssey by Greff et al., 2015 Marco Lippi Machine Learning 39 / 80

40 Recursive Neural Networks (RecNNs) A generalization of RNNs to handle structured data in the form of a dependency graph such as a tree [Figure by Socher et al., 2014] RNNs can be seen as RecNNs having a linear chain structure. Marco Lippi Machine Learning 40 / 80

41 Recursive Neural Networks (RecNNs) Exploit compositionality (e.g., parse trees in text) [Figure by R. Socher] Marco Lippi Machine Learning 41 / 80

42 Recursive Neural Networks (RecNNs) Exploit compositionality (e.g., object parts in images) [Figure by Socher et al., 2011] Marco Lippi Machine Learning 42 / 80

43 Recursive Neural Tensor Networks (RNTNs) Recently (2013) proposed by Stanford for sentiment analysis composition function over sentence parse tree exploit parameters (tensors) that are common to all nodes (tensor) backpropagation through structure Later ( ) also a tree-structured version of LSTMs Marco Lippi Machine Learning 43 / 80

44 Recursive Neural Tensor Networks (RNTNs) Sentiment Treebank built by crowdsourcing 11,855 sentences from movie reviews 215,154 labeled phrases (sub-sentences) [Figure by Socher et al., 2014] Marco Lippi Machine Learning 44 / 80

45 Recursive Neural Tensor Networks (RNTNs) [Figure by Socher et al., 2014] Marco Lippi Machine Learning 45 / 80

46 Recursive Neural Tensor Networks (RNTNs) [Figure by Socher et al., 2014] Marco Lippi Machine Learning 46 / 80

47 Recursive Neural Tensor Networks (RNTNs) [Figure by Socher et al., 2014] Marco Lippi Machine Learning 47 / 80

48 Recursive Neural Tensor Networks (RNTNs) [Figure by Socher et al., 2014] Marco Lippi Machine Learning 48 / 80

49 Recursive Neural Tensor Networks (RNTNs) Very nice model but: not easy to adapt it to other domains (Reddit, Twitter, etc.) need to compute parse tree in advance need classification at phrase-level (expensive) Marco Lippi Machine Learning 49 / 80

50 Other applications Marco Lippi Machine Learning 50 / 80

51 Modeling speech Several tasks... Speech recognition Speaker identification Speaker gender classification Phone classification Music genre classification Artist classification Music retrieval Marco Lippi Machine Learning 51 / 80

52 Modeling speech Feature extraction from acoustic signals Tons of unsupervised data! [Figure by H. Lee] Marco Lippi Machine Learning 52 / 80

53 Modeling speech [Figure by H. Lee] Marco Lippi Machine Learning 53 / 80

54 Multimodal Deep Learning [Figure by Ngiam et al.] Can we gain something from shared representations...?... Most probably yes! Marco Lippi Machine Learning 54 / 80

55 Other applications: Bioinformatics, Neuroscience, etc... Protein structure prediction (contact maps) Deep Spatio-Temporal Architectures [Di Lena et al., 2012] Sequence: ASCDEVVGSACH...CPPGAERMMAYGV [Figure by Rafferty et al.] Marco Lippi Machine Learning 55 / 80

56 Other applications: Bioinformatics, Neuroscience, etc... Predicting the aqueous solubility of drug-like molecules Recursive neural networks [Lusci et al., 2013] [Figure by Lusci et al.] Marco Lippi Machine Learning 56 / 80

57 Ongoing Research and Concluding Remarks Marco Lippi Machine Learning 57 / 80

58 Summary Deep learning has obtained breakthrough results in many tasks exploit unsupervised data learning feature hierarchies some optimization tricks (dropout, rectification,...) Is this the solution to all AI problems? Probably not but... for certain types of task it is hard to compete huge datasets and many computational resources big companies will likely play the major role huge space for applications upon deep learning systems...but what is missing? Marco Lippi Machine Learning 58 / 80

59 A nice overview of methods Marco [Figure Lippi by M.A. Machine Ranzato] Learning 59 / 80

60 We are not yet there... Still far from human-level in unrestricted domains... Marco Lippi Machine Learning 60 / 80

61 Again: symbolic vs. sum-symbolic Still basically a sub-symbolic approach Representation learning: a step towards symbols... What are the connections with symbolic approaches? What about logic and reasoning? Some steps forward: babi Facebook Neural Conversational Google Neural Turing Machines GoogleDeepMind Marco Lippi Machine Learning 61 / 80

62 babi tasks et al. (Facebook) Text understanding and reasoning with deep networks The (20) babi tasks The Children s Book Test The Movie Dialog dataset The SimpleQuestions dataset Marco Lippi Machine Learning 62 / 80

63 babi tasks (Facebook) [Table by Weston et al.] Marco Lippi Machine Learning 63 / 80

64 babi tasks (Facebook) [Table by Weston et al.] Marco Lippi Machine Learning 64 / 80

65 Children s Book Test [Table by Hill et al., 2016] Marco Lippi Machine Learning 65 / 80

66 Movie Dialog dataset [Table by Dodge et al., 2016] Marco Lippi Machine Learning 66 / 80

67 SimpleQuestions dataset [Table by Bordes et al., 2015] Marco Lippi Machine Learning 67 / 80

68 Neural Conversational Model (Google) [Table by Vinyalis & Le, 2015] Marco Lippi Machine Learning 68 / 80

69 Neural Conversational Model (Google) [Table by Vinyalis & Le, 2015] Marco Lippi Machine Learning 69 / 80

70 Neural Turing Machines (Google Deepmind) [Figure by Graves et al., 2014] Inspired by Turing Machines Memory (tape) to read/write through deep networks Capable of learning simple algorithms (e.g., sorting) Marco Lippi Machine Learning 70 / 80

71 Learning concepts Examples from the ImageNet category Restaurant Is it enough to learn the concept of a restaurant? Marco Lippi Machine Learning 71 / 80

72 Software packages and a few references Marco Lippi Machine Learning 72 / 80

73 Tensorflow Developed by Google Python Computational graph abstraction Parallelize over both data and model The multi-machine part is not open source Nice tool to visualize stuff (Tensorboard) A little slower than other systems Provides only a few (one?) pre-trained models Marco Lippi Machine Learning 73 / 80

74 Theano University of Montréal (Yoshua Bengio s group) Python Computational graph abstraction The code is somehow difficult (low-level) Long compile times Easily GPU and multi-gpu Marco Lippi Machine Learning 74 / 80

75 Keras and Lasagne Keras is a wrapper for Theano or Tensorflow Python Plug-and-play layers, losses, optimizers, etc. Sometimes error messages can be criptic... Lasagne is a wrapper for Theano Python Plug-and-play layers, losses, optimizers, etc. Model zoo with plenty pre-trained architectures Still employs some symbolic computation as plain Theano Marco Lippi Machine Learning 75 / 80

76 Torch Facebook, Google, Twitter, IDIAP,... mostly written in Lua and C sharing similarities to python (e.g. tensors vs. numpy arrays) module nn to train neural networks same code running for both CPU and GPU One notable product by Torch: Overfeat A CNN for image classification, object detection, etc. Marco Lippi Machine Learning 76 / 80

77 Caffe Berkeley Vision and Learning Center (BVLC) written in C++ Python and MATLAB bindings (although not much documented) very popular for CNNs, not much for RNNs... quite a large model zoo (AlexNet, GoogLeNet, ResNet,... ) scripts for training without writing code Just to give an idea of Caffe s performance... During training 60M images per day with a single GPU At test time 1 ms/image Marco Lippi Machine Learning 77 / 80

78 Matlab Plenty of resources Code by G. Hinton on RBMs and DBNs (easy to try!) Autoencoders (many different implementations) Convolutional neural networks... Marco Lippi Machine Learning 78 / 80

79 Websites and references Introductory paper by Yoshua Bengio: Learning Deep Architectures for AI Marco Lippi Machine Learning 79 / 80

80 Videos Geoffrey Hinton on DBNs: Yoshua Bengio (Deep Learning lectures): Yann LeCun (CNNs, energy-based models): Marco Lippi Machine Learning 80 / 80