Backpropagation. Deep Learning Theory and Applications. Kevin Moon Guy Wolf

Size: px

Start display at page:

Download "Backpropagation. Deep Learning Theory and Applications. Kevin Moon Guy Wolf"

Cori Anthony
6 years ago
Views:

1 Deep Learning Theory and Applications Backpropagation Kevin Moon Guy Wolf CPSC/AMTH 663

2 Calculating the gradients We showed how neural networks can learn weights and biases Gradient descent/stochastic gradient descent How do we calculate the gradients at each node in each layer? Answer: Backpropagation!

3 Backpropagation history Introduced in 1970 s Unappreciated until 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Wiiams Described several neural networks where backpropagation works far faster than earlier learning approaches Today, backpropagation is the workhorse of learning neural networks

4 Why study backpropagation? Why not treat backpropagation like a black box? Reason: to obtain understanding Backpropagation provides an expression for, the partial derivative of the cost function CC wrt any weight ww (or bias bb) I.e., backpropagation tes us how quickly the cost changes when changing weights and biases backpropagation gives us detailed insights into how changing the weights and biases changes the overa network

5 Outline 1. Preliminaries Matrix multiplication for computing applications Cost function assumptions Hadamard product Error at the node 2. Four fundamental equations of backpropagation The equations Proofs 3. The backpropagation algorithm 4. Is backpropagation fast? 5. Backpropagation: the big picture

6 Preliminaries CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation

Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation ww jjjj = the weight for the connection from the kkth

7 Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation ww jjjj = the weight for the connection from the kkth neuron in the ( 1)th layer to the jjth neuron in the th layer Question: why is jj the output neuron and kk the input neuron? Answer: it simplifies the matrix notation

8 Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation bb jj = bias of the jjth neuron in the th layer aa jj = activation (output) of the jjth neuron in the th layer

9 Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation The activation aa jj of the jjth neuron the th layer is related to the activations in the ( 1)th layer: aa jj = σσ kk ww jjjj aa 1 kk + bb jj

10 Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation aa jj = σσ Rewrite in matrix form: ww = a weight matrix for layer kk ww jjjj aa 1 kk + bb jj Entries are the weights connecting to the th layer of neurons; i.e. the entry in the jjth row and kkth column is ww jjjj Bias vector bb for the th layer (bb jj in the jjth component) Activation vector aa for the th layer (aa jj in the jjth component)

11 Matrix multiplication for computing activations Vectorizing a function Apply the function element-wise I.e., components of σσ(vv) are σσ vv jj = σσ(vv jj ) Example: ff xx = xx 2 Vectorized form of ff has the foowing effect: ff 2 3 = ff(2) ff(3) = 4 9 Activation computation aa = σσ ww aa 1 + bb

12 Matrix multiplication for computing activations CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation Activation computation aa = σσ ww aa 1 + bb Provides a global view of layer-layer relationships Apply weight matrix to activations, add bias vector, then apply σσ Easier and more succinct than neuron-by-neuron view Matrix and vector computations are fast We wi also use an intermediate quantity: zz = ww aa 1 + bb zz is the weighted input to the neurons in layer aa = σσ zz zz jj is the weighted input to the activation function for neuron jj in layer

13 Cost function assumptions Goal of backpropagation: compute the partial derivatives and of the cost function CC to any weight ww or bias bb in the network Example cost function: quadratic cost CC = 1 2nn yy xx aa LL xx 2 xx Sum is over training examples xx yy(xx) is the corresponding desired output aa LL (xx) is the vector of activation output of the network when xx is input (LL denotes the number of layers in the network)

14 Cost function assumptions Example cost function: quadratic cost CC = 1 2nn yy xx aa LL xx 2 xx Assumption 1: The cost function can be written as an average CC = 1 nn xx CC xx over cost functions CC xx for individual training examples xx. CC xx = 1 yy 2 aall 2 for quadratic cost Reason: backpropagation can calculate partial derivatives CC xx and CC xx for a single training example and recovered by averaging over training examples For notational simplicity, we assume xx is fixed and drop the subscript (i.e., write CC xx as CC for now)

15 Cost function assumptions Assumption 2: Cost can be written as a function of the neural network outputs For quadratic cost: CC = 1 2 yy aall 2 = 1 2 jj yy ii aa jj 2

16 Cost function assumptions Assumption 1: The cost function can be written as an average CC = 1 nn xx CC xx over cost functions CC xx for individual training examples xx. Assumption 2: Cost can be written as a function of the neural network outputs.

17 Hadamard product Let ss and tt be vectors with same dimension ss tt is the elementwise product of the vectors ss tt jj = ss jj tt jj Example: = = 3 8 Referred to as the Hadamard product or Schur product

18 Error at the node CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation Backpropagation helps us understand how changing weights and biases in a network changes the cost function I.e., backpropagation gives us the partial derivatives We first introduce an intermediate quantity δδ jj to ww jjjj ww jjjj and bb jj δδ jj is the error in the jjth neuron and the th layer Backpropagation enables us to compute δδ jj which we wi relate and bb jj

19 Error at the node As input comes in, the demon adds a little change zz jj to the neuron s weighted input Output becomes σσ zz jj + zz jj Change propagates through later layers causing zz jj zz jj change in cost

20 Error at the node Change zz jj results in zz jj zz jj change in cost Case 1: zz jj is large (either pos. or neg.) Demon can lower cost a lot by choosing zz jj with opposite sign of zz jj Case 2: zz jj is close to zero Can t decrease cost much by perturbing zz jj

21 Error at the node Case 1: zz jj is large (either pos. or neg.) Demon can lower cost a lot by choosing zz jj with opposite sign of zz jj Case 2: zz jj is close to zero Can t decrease cost much by perturbing zz jj zz jj is a measure of error in this heuristic sense Define δδ jj = zz jj (δδ is the vectorized form) Similar results obtained by considering aa jj

22 Four fundamental equations of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation

23 First fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation Error in the output layer: δδ jj LL = aa jj LL σσ zz jj LL aa jj LL measures how fast cost CC changes as function of jjth output Example: if CC doesn t depend much on neuron jj, then δδ jj LL wi be sma σσ zz LL jj measures how fast the activation function σσ LL changes at zz jj

24 First fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation Error in the output layer: δδ jj LL = aa jj LL σσ zz jj LL A parts are easily computable zz jj LL is computed while computing behavior of the network σσ zz jj LL foows easily aa jj LL depends on cost function Quadratic cost: CC = 1 2 jj yy jj aa jj LL 2 aa jj LL = aa jj LL yy jj

25 First fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation Error in the output layer: δδ jj LL = aa jj LL σσ zz jj LL Matrix-based form: δδ LL = aa CC σσ zz LL aa CC jj = LL aa jj Example: quadratic cost δδ LL = aa LL yy σσ zz LL Easily computed using Numpy

26 Second fundamental equation of backpropagation Error in the th layer in terms of the error in the next layer: δδ = ww +1 TT δδ +1 σσ (zz ) Interpretation Suppose δδ +1 is known Applying ww +1 TT moves the error backward through the network Gives a measure of the error at the output of the th layer Taking the Hadamard product σσ (zz ) moves the error backward through the activation function in layer Gives the error δδ in the weighted input to layer

27 Second fundamental equation of backpropagation First fundamental equation δδ LL = aa CC σσ zz LL Second fundamental equation δδ = ww +1 TT δδ +1 σσ (zz ) Can compute the error δδ for any error using these equations 1. Compute δδ LL 2. Compute δδ LL 1 3. Compute δδ LL 2 4. And so on a the way back through the network

28 Third fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation The rate of change of the cost wrt any bias in the network bb jj = δδ jj Easily computed from previous equations

29 Fourth fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation The rate of change of the cost wrt any weight in the network Easily computed ww jjjj = aa kk 1 δδ jj Simplified notation: = aa in δδ out aa in is the activation of the neuron input to the weight ww δδout is the error of the neuron output from the weight ww

30 Fourth fundamental equation of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation The rate of change of the cost wrt any weight in the network = aa in δδ out If aa in 0, the gradient wi be sma The weight learns slowly (i.e. doesn t change much during gradient descent) I.e., weights output from low-activation neurons learn slowly

31 Insights from the 4 fundamental equations Consider the output layer: δδ jj LL = aa jj LL σσ zz jj LL σσ becomes very flat when σσ 0 or 1 σσ zz jj LL 0 weight in final layer wi learn slowly if output neuron is saturated (i.e. 0 or 1)

32 Insights from the 4 fundamental equations Similar insights for other layers: Weights wi learn slowly if either the input neuron is low activation or the output neuron has saturated (i.e. low or high activation) The 4 fundamental equations hold for any activation functions We can use these equations to design activation functions E.g., choose σσ s.t. σσ is always positive and never close to zero (we see this later in the course) Understanding the 4 fundamental equations can guide us designing neural networks

33 The 4 fundamental equations of backpropagation CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation 1. δδ LL = aa CC σσ zz LL 2. δδ = ww +1 TT δδ +1 σσ (zz ) bb jj = δδ jj ww jjjj = aa kk 1 δδ jj

34 Proofs of the 4 fundamental equations of backpropagation A are based on the chain rule from multivariable calculus

35 Proof of the 1 st equation By definition: δδ jj LL = zz jj LL Use multivariate chain rule to obtain: δδ LL jj = LL aa jj LL LL aa jj zz jj Reca that aa jj LL = σσ zz jj LL δδ jj LL = aa jj LL σσ zz jj LL

36 Proof of the 2 nd equation Rewrite δδ jj in terms of δδ kk +1 using chain rule δδ jj = Differentiating gives zz kk +1 zz jj kk zz kk +1 zz jj δδ kk +1 = ww +1 kkkk σσ (zz jj ) δδ jj = kk ww +1 kkkk δδ +1 kk σσ (zz jj )

37 Proof of the 3 rd equation Use multivariate chain rule to obtain: bb = zz jj jj zz jj bb jj zz jj bb jj = 1 bb jj = δδ jj

38 Proof of the 4 th equation Use multivariate chain rule to obtain: = zz jj zz jj ww jjkk ww jjkk zz jj ww jjjj = aa kk 1 ww jjjj = aa kk 1 δδ jj

39 The backpropagation algorithm CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation

40 The backpropagation algorithm Backpropagation equations provide a way for computing the gradient of the cost function 1. Input xx: Set the activation aa 1 for the input layer 2. Feedforward: For each = 2,3,, LL, compute zz = ww aa 1 + bb and aa = σσ zz 3. Output error δδ LL : Compute δδ LL = aa CC σσ zz LL 4. Backpropagate the error: For each = LL 1, LL 2,, 2 compute δδ = ww +1 TT δδ +1 σσ (zz ) 5. Output: The cost function gradient is and bb jj = δδ jj ww jjjj = aa 1 δδ jj

41 Backpropagation with SGD Backpropagation computes the gradient of the cost function for a single training example CC = CC xx Typicay, backpropagation is combined with a learning algorithm such as stochastic gradient descent

42 Backpropagation with SGD 1. Input a set of training examples 2. For each training example xx: Set the input activation aa xx,1 and do the foowing: Feedforward: For each = 2,3,, LL compute zz xx, = ww aa xx, 1 + bb and aa xx, = σσ zz xx, Output error δδ xx,ll : Compute δδ xx,ll = aa CC σσ zz xx,ll Backpropagate the error: For each = LL 1, LL 2,, 2 compute δδ xx, = ww +1 TT δδ xx,+1 σσ (zz xx, ) 3. Gradient descent: for each = LL, LL 1,, 2 update weights ww ww ηη mm xx δδ xx, aa xx, 1 TT and the biases bb bb ηη mm xx δδ xx,

43 Is backpropagation fast? CPSC/AMTH 663 (Kevin Moon/Guy Wolf) Backpropagation

44 Is backpropagation fast? Consider an alternative approach Suppose you try to not use the chain rule and regard the cost as a function of the weights directly CC = CC(ww) Could use the approximation: CC ww + εεee jj CC(ww) ww jj εε > 0 is a sma positive number ee jj = unit vector in jjth direction I.e., we re estimating the derivatives directly Same idea applies to biases Advantage: very easy to implement Disadvantage: very slow εε

45 Is backpropagation fast? Could use the approximation: CC ww + εεee jj ww jj εε Why is it slow? CC(ww) Suppose we have a miion weights in our network For each weight ww jj we compute CC ww + εεee jj Thus we need to compute the cost function a miion times, requiring a miion forward passes through the network per training sample In contrast, backpropagation computes a the partial derivatives using one forward and backward pass ww jj Roughly equivalent to two forward passes miion passes

46 Backpropagation: the big picture Can we build more intuition? How could someone have discovered backpropagation?

47 Improved intuition Suppose we make a sma change ww jjjj to weight ww jjjj

48 Improved intuition Suppose we make a sma change ww jjjj to weight ww jjjj This causes a change in the output activation of the corresponding neuron

49 Improved intuition Suppose we make a sma change ww jjjj to weight ww jjjj This causes a change in the output activation of the corresponding neuron This causes a change in a activations in the next layer

50 Improved intuition Suppose we make a sma change ww jjjj to weight ww jjjj This causes a change in the output activation of the corresponding neuron This causes a change in a activations in the next layer And so on until the final layer

51 Improved intuition Change in cost CC is related to the change ww jjjj : CC ww jjjj Possible approach to calculate ww jjjj ww jjjj change in ww jjjj propagates to change CC Use easily computable quantities along the way is to track how a sma

52 Improved intuition Relate change in weight ww jjjj aa jj aa jj ww jjjj to change in activation aa jj ww jjjj Change in activation aa jj causes changes in a activations of the ( + 1)th layer Focus on aa qq +1 :

53 Improved intuition Focus on aa qq +1 : aa qq +1 aa qq +1 aa jj aa qq +1 aa qq +1 aa jj aa jj aa jj ww jjjj ww jjjj

54 Improved Intuition Extending this to multiple layers gives: CC aall mm LL LL 1 aa mm aa aa +1 qq nn aa jj aa jj ww jjjj ww jjjj Represents the change in CC due to a change in ww jjjj through one path in the network A paths: CC mmmm qq aa mm LL aa mm LL aa nn LL 1 aa qq +1 aa jj aa jj ww jjjj ww jjjj

55 Improved Intuition = ww jjjj Interpretation mmmm qq aa mm LL aa mm LL aa nn LL 1 aa qq +1 aa jj aa jj ww jjjj Every edge between two neurons is associated with a rate factor (the partial derivative of one neuron s activation wrt the other neuron s activation) The rate factor for a path is the product of rate factors along the path Total rate of change is the sum of the rate factors of a paths from the initial weight to final cost

56 Improved Intuition Single path 1. Could derive expressions for each partial derivative Use calculus 2. Write sums as matrix multiplications 3. Simplify Result is backpropagation

57 Discovering backpropagation 1. Foow the long approach just described 2. Discover there are some obvious simplifications 3. Make those simplifications to get a shorter proof 4. Repeat until you get the nice proof we did in class

58 Further reading Nielsen book, chapter 2 Goodfeow et al., section 6.5

WRITTEN PRELIMINARY Ph.D. EXAMINATION. Department of Applied Economics. January 28, Consumer Behavior and Household Economics.

WRITTEN PRELIMINARY Ph.D. EXAMINATION Department of Applied Economics January 28, 2016 Consumer Behavior and Household Economics Instructions Identify yourself by your code letter, not your name, on each