Applications of Neural Networks

Transcription

1 Applications of Neural Networks MPhil ACS Advanced Topics in NLP Laura Rimell 25 February

2 NLP Neural Network Applications Language Models Word Embeddings Tagging Parsing Sentiment Machine Translation A neural network dream (Google Research) 2

3 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 3

4 Neuron input variables x 1 x 2 neuron h non-linear activation function output x 3 x 4 w yh w h5 y = w f( w x + b) y x 5 yh hi i b 4

5 Neural Network input layer hidden layer output layer x 1 x 2 w 11 w 21 y 1 v 11 v 21 x 3 y 2 x 4 x 5 w 35 v 33 y 3 b y = V f(w x + b) YH HX 5

6 Neural Network W HX V YH input layer hidden layer output layer x h = f(w x) y =Vh HX 6

7 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 7

8 Deep Feed-Forward Network W 1 W 2 W 3 W 4 W 5 8

9 Output Layers vector softmax y = k 3 l = 1 e w x i ki i e w x i li i 9

10 Backpropagation Δw = -α δe δw mean squared error 1 2 E = (t - y ) n k k w Gradient descent 1 E = - t ln(y ) + (1-t )ln(y ) n k k k k cross-entropy SGD, Adagrad, Adadelta,... 10

11 Feed-Forward Network Limitations Fixed-size input vector Fixed-size output vector Fixed number of computational steps (layers) 11

12 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 12

13 Recursive Neural Network (RecNN) One weight matrix (simplest version) f(w(x1 x2)) W H2X 13

14 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 14

15 Recurrent Neural Network o(t) = g(vs(t)) V YH output layer o(t) hidden layer s(t) (state) s(t) = f(ux(t) + Ws(t-1)) W HH input layer x(t) U HX 15

16 Unfolded RNN o(t-1) o(t) o(t+1) V s(t-1) W V s(t) W V s(t+1) U U U x(t-1) x(t) x(t+1) 16

17 Backpropagation Through Time o(t-1) o(t) w s(t) Δw = -α δe δw x(t-1) x(t) Problem: vanishing gradients

18 RNN Language Model As seen in Word Embeddings topic Bengio et al.,

19 Long Short Term Memory (LSTM) Preserves error to help with vanishing gradients Gate to keep or discard information Each cell has own set of learned weights 19

20 Gated Recurrent Unit (GRU) Chung et al., arxiv

21 Outline Neural network basics NN architectures Feedforward networks and backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 21

22 Tagging The cat sat on the mat. DET NN1 VBD PRP DET NN1 PUNCT NP/N N S[dcl]\NP (S[dcl]\NP)/NP NP/N N PUNCT B_NP I_NP B_VP B_PP B_NP I_NP O 22

23 Tagging with Feed-Forward Network Collobert et al., JMLR 2011 Lewis & Steedman, EMNLP

24 Tagging with RNN Task: detect and tag Direct Subjective Expressions, Expressive Subjective Expressions Irsoy & Cardie, EMNLP

25 Deep Bidirectional RNN Bidirectional RNN incorporates info from preceding and following words h = [h; h] represents past and future around a word Deep (stacked) RNN Irsoy & Cardie, EMNLP

26 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation Back to architectures for MT Encoder-Decoder Networks 26

27 Constituent Parsing

28 Constituent Parsing with Feed-Forward NN Neural embeddings replace sparse features in CRF parser Durret & Klein, ACL

29 Constituent Parsing with RecNN Used for re-ranking n-best PCFG parser output Socher et al., ACL

30 Dependency Parsing Chen & Manning, EMNLP

31 Dependency Parsing with Feed-Forward NN Chen & Manning, EMNLP

32 Dependency Parsing with RNN (v1) Yazdani & Henderson, CoNLL

33 Dependency Parsing with RNN (v2) Stack LSTM: adds stack pointer Dyer et al., ACL

34 Dependency Parsing with RNN (v2) sentence: an overhasty decision was made Dyer et al., ACL

35 Outline Neural network basics NN architectures Feedforward Networks and Backpropagation Recursive Neural Networks Recurrent Neural Networks Applications Tagging Parsing Machine Translation and Encoder-Decoder Networks 35

36 Machine Translation 36

37 Encoder-Decoder Network (RNN) (gen l idea) Example from Richard Socher lecture 37

38 Encoder-Decoder Network (RNN) (v1) Used for scoring phrase pairs in phrase table of standard SMT system Cho et al., EMNLP

39 Encoder-Decoder Network (RNN) (v2) Used for direct translation with beam search decoder 4-layer deep LSTM Input words in reverse order Sutskever et al., ANIPS

40 NLP Neural Network Applications 40