Backpropagation. CS 478 Backpropagation 1

Transcription

1 Backpropagation CS 478 Backpropagation 1

2 Mutiayer Nets? Linear Systems F(cx) = cf(x) F(x+y) = F(x) + F(y) I N M Z Z = (M(NI)) = (MN)I = PI CS 478 Backpropagation 2

3 Eary Attempts Committee Machine Randomy Connected Vote Taking TLU (Adaptive) (non-adaptive) Majority Logic "Least Perturbation Principe" For each pattern, if incorrect, change just enough weights into interna units to give majority. Choose those cosest to their threshod (LPP & changing undecided nodes) CS 478 Backpropagation 3

4 Perceptron (Frank Rosenbatt) Simpe Perceptron S-Units A-units R-units (Sensor) (Association) (Response) Random to A-units fixed weights adaptive Variations on Deta rue earning Why S-A units? CS 478 Backpropagation 4

5 Backpropagation Rumehart (1986), Werbos (74),, exposion of neura net interest Muti-ayer supervised earning Abe to train muti-ayer perceptrons (and other topoogies) Uses differentiabe sigmoid function which is the smooth (squashed) version of the threshod function Error is propagated back through earier ayers of the network Very fast efficient way to compute gradients! CS 478 Backpropagation 5

6 Muti-ayer Perceptrons trained with BP Can compute arbitrary mappings Training agorithm ess obvious First of many powerfu muti-ayer earning agorithms CS 478 Backpropagation 6

7 Responsibiity Probem Output 1 Wanted 0 CS 478 Backpropagation 7

8 Muti-Layer Generaization CS 478 Backpropagation 8

9 Mutiayer nets are universa function approximators Input, output, and arbitrary number of hidden ayers 1 hidden ayer sufficient for DNF representation of any Booean function - One hidden node per positive conjunct, output node set to the Or function 2 hidden ayers aow arbitrary number of abeed custers 1 hidden ayer sufficient to approximate a bounded continuous functions 1 hidden ayer was the most common in practice, but recenty Deep networks show exceent resuts! CS 478 Backpropagation 9

10 z n 1 n 2 x 1 x 2 (0,1) (1,1) (0,1) (1,1) x 2 x 2 (0,0) x 1 (1,0) (0,0) x 1 (1,0) (0,1) (1,1) n 2 (0,0) n 1 (1,0) CS 478 Backpropagation 10

11 Backpropagation Muti-ayer supervised earner Gradient descent weight updates Sigmoid activation function (smoothed threshod ogic) Backpropagation requires a differentiabe activation function CS 478 Backpropagation 11

12 CS 478 Backpropagation 12

13 Muti-ayer Perceptron (MLP) Topoogy i k i j k i k i Input Layer Hidden Layer(s) Output Layer CS 478 Backpropagation 13

14 Backpropagation Learning Agorithm Unti Convergence (ow error or other stopping criteria) do Present a training pattern Cacuate the error of the output nodes (based on T - Z) Cacuate the error of the hidden nodes (based on the error of the output nodes which is propagated back to the hidden nodes) Continue propagating error back unti the input ayer is reached Then update a weights based on the standard deta rue with the appropriate error function δ Δw ij = C δ j Z i CS 478 Backpropagation 14

15 Activation Function and its Derivative Node activation function f(net) is typicay the sigmoid Z j = f ( net j 1 ) = 1+ e net j Derivative of activation function is a critica part of the agorithm f '(net j ) = Z j (1 Z j ) Net Net CS 478 Backpropagation 15

16 CS 478 Backpropagation 16 j k k k i i i i Backpropagation Learning Equations Node] [Hidden ) ( ' ) ( [Output Node] ) ( ' ) ( j k jk k j j j j j i j ij net f w net f Z T Z C w = = = Δ δ δ δ δ

17 Network Equations Output: O j = f(net j ) = 1 1+e-net j f'(net j ) = O j net j = O j (1 - O j ) w ij (genera node): C O i j w ij (output node): j = (t j - O j ) f'(net j ) w ij = C O i j = C O i (t j - O j ) f'(net j ) w ij (hidden node) j = ( k w jk ) f'(net j ) k w ij = C O i j = C O i ( ( k w jk ) ) f'(net j ) k CS 478 Backpropagation 17

18 BP-1) A backpropagation mode has initia weights as shown. Work through one cyce of earning for the f oowing pattern(s). Assume 0 momentum and a earning constant of 1. Round cacuations to 3 significant digits to the right of the decima. Give vaues for a nodes and inks for activation, output, error signa, weight deta, and fina weights. Nodes 4, 5, 6, and 7 are just input nodes and do not have a sigmoida output. For each node cacuate the foowing (show necessary equati on for each). Hint: Cacuate bottom-top-bottom. a = o = = w = w = a) A weights initiay 1.0 Training Patterns 1) 0 0 -> 1 2) 0 1 -> CS 478 Backpropagation 18

19 BP-1) net2 = wi xi = (1*0 + 1*0 + 1*1) = 1 net3 = 1 o2 = 1/(1+e -net ) = 1/(1+e -1 ) = 1/(1+.368) =.731 o3 =.731 o4 = 1 net1 = (1* * ) = o1 = 1/(1+e )= = (t1 - o1) o1 (1 - o1) = ( ).921 ( ) = w21 = j oi = 1 o2 = 1 * *.731 = w31 = 1 * *.731 = w41 = 1 * * 1 = = oj (1 - oj) k wjk = o2 (1 - o2) 1 w21 =.731 (1-.731) ( * 1) = = w52 = j oi = 2 o5 = 1 * * 0 = 0 w62 = 0 w72 = 1 * * 1 = w53 = 0 w63 = 0 w73 = 1 * * 1 = CS 478 Backpropagation +1 19

20 Backprop Homework For your homework update the weights for the second pattern of the training set 0 1 -> 0 And then go to ink beow: Neura Network Payground using the tensorfow too and pay around with the BP simuation. Try different training sets, ayers, inputs, etc. and get a fee for what the nodes are doing. You do not have to hand anything in for this part. CS 478 Backpropagation 20

21 Inductive Bias & Intuition Node Saturation - Avoid eary, but a right ater When saturated, an incorrect output node wi sti have ow error Start with weights cose to 0 Saturated error even when wrong? Mutipe TSS drops Not exacty 0 weights (can get stuck), random sma Gaussian with 0 mean Can train with target/error detas (e.g..1 and.9 instead of 0 and 1) Intuition Manager/Worker Interaction Gives some stabiity Inductive Bias Start with simpe net (sma weights, initiay inear changes) Smoothy buid a more compex surface unti stopping criteria CS 478 Backpropagation 21

22 Muti-ayer Perceptron (MLP) Topoogy i k i j k i k i Input Layer Hidden Layer(s) Output Layer CS 478 Backpropagation 22

23 Loca Minima Most agorithms which have difficuties with simpe tasks get much worse with more compex tasks Good news with MLPs Many dimensions make for many descent options Loca minima more common with very simpe/toy probems, very rare with arger probems and arger nets Even if there are occasiona minima probems, coud simpy train mutipe nets and pick the best Some agorithms add noise to the updates to escape minima CS 478 Backpropagation 23

24 Loca Minima and Neura Networks Neura Network can get stuck in oca minima for sma networks, but for most arge networks (many weights), oca minima rarey occur in practice This is because with so many dimensions of weights it is unikey that we are in a minima in every dimension simutaneousy amost aways a way down CS 312 Approximation 24

25 Stopping Criteria and Overfit Avoidance SSE Epochs Vaidation/Test Set Training Set More Training Data (vs. overtraining - One epoch imit) Vaidation Set - save weights which do best job so far on the vaidation set. Keep training for enough epochs to be fairy sure that no more improvement wi occur (e.g. once you have trained m epochs with no further improvement, stop and use the best weights so far, or retrain with a data). Note: If using N-way CV with a vaidation set, do n runs with 1 of n data partitions as a vaidation set. Save the number i of training epochs for each run. To get a fina mode you can train on a the data and stop after the average number of epochs, or a itte ess than the average since there is more data. Specific techniques for avoiding overfit Less hidden nodes not a great approach because may underfit Weight decay, Jitter, Error detas, Dropout (discuss with ensembes) CS 478 Backpropagation 25

26 Vaidation Set - ML Manager Often you wi use a vaidation set (separate from the training or test set) for stopping criteria, etc. In these cases you shoud take the vaidation set out of the training set which has aready been aocated by the ML manager. For exampe, you might use the random test set method to randomy break the origina data set into 80% training set and 20% test set. Independent and subsequent to the above routines you woud take n% of the training set to be a vaidation set for that particuar training exercise. CS Backpropagation 26

27 Learning Rate Learning Rate - Reativey sma ( common), if too arge BP wi not converge or be ess accurate, if too sma is sower with no accuracy improvement as it gets even smaer Gradient ony where you are, too big of jumps? CS 478 Backpropagation 27

28 Learning Rate

29 Momentum Simpe speed-up modification (type of adaptive earning rate) Δw(t+1) = Cδ x i + αδw(t) Weight update maintains momentum in the direction it has been going Faster in fats Coud eap past minima (good or bad) Significant speed-up, common vaue α.9 Effectivey increases earning rate in areas where the gradient is consistenty the same sign. (Which is a common approach in adaptive earning rate methods). These types of terms make the agorithm ess pure in terms of gradient descent. However Not a big issue in overcoming oca minima Not a big issue in entering bad oca minima CS 478 Backpropagation 29

30 Hidden Nodes Typicay one fuy connected hidden ayer. Common initia number is 2n or 2ogn hidden nodes where n is the number of inputs In practice train with a sma number of hidden nodes, then keep doubing, etc. unti no more significant improvement on test sets A output and hidden nodes shoud have bias weights Hidden nodes discover new higher order features which are fed into the output ayer Zipser - Linguistics i k Compression i j k i k i CS 478 Backpropagation 30

31 Hyperparameter Seection LR (e.g..1) Momentum (.5.99) Connectivity: typicay fuy connected between ayers Number of hidden nodes: Too many nodes make earning sower, coud overfit Too many hidden nodes is usuay OK if using a reasonabe stopping criteria Too few wi underfit Number of ayers: 1 (common) or 2 hidden ayers which are usuay sufficient for good resuts, attenuation makes earning very sow modern deep earning approaches show significant improvement using many ayers Manua CV common method to set hyperparameters: tria and error runs Often sequentia: find one hyperparameter vaue with others hed constant, freeze it, find next hyperparameter, etc. Can aso do an automated search: Grid, Random, Bayesian Random is empiricay most consistenty effective typicay each hyperparameter is chosen with a uniform distribution from a og scae for each tria Hyperparameters coud be earned by the earning agorithm in which case you must take care to not overfit the training data CS 478 Backpropagation 31

32 BP Lab Go over Lab together CS 478 Backpropagation 32

33 Debugging your ML agorithms Debugging ML agorithms is difficut Unsure beforehand about what the resuts shoud be Adaptive agorithm can earn to compensate somewhat for bugs (sign wrong on bias update, etc.) Bugs in accuracy evauation code common fase hopes! Do a sma exampe by hand (e.g. your homework) and make sure your agorithm gets the exact same resuts (and accuracy) Compare resuts with our suppied snippets from in earning suite Compare resuts (not code, etc.) with cassmates Compare resuts with a pubished version of the agorithms (e.g. WEKA), won t be exact because of different training/test spits, etc. Use Zarndt s thesis (or other pubications) to get a bapark fee of how we you shoud expect to do on different data sets. 33

34 Batch Update With On-ine (stochastic) update we update weights after every pattern With Batch update we accumuate the changes for each weight, but do not update them unti the end of each epoch Batch update gives a correct direction of the gradient for the entire data set, whie on-ine coud do some weight updates in directions quite different from the average gradient of the entire data set Based on noisy instances and aso just that specific instances wi not represent the average gradient Proper approach? - Conference experience Most (incuding us) assumed batch more appropriate, but batch/onine a non-critica decision with simiar resuts We show that batch is ess efficient more in 678 CS 478 Backpropagation 34

35 Mutipe Outputs Typica to have mutipe output nodes, even with just one output feature (e.g. Iris data set) Woud if there are mutipe "independent output features" Coud train independent networks Aso common to have them share hidden ayer May find shared features Transfer Learning Coud have shared and separate subsequent hidden ayers, etc. Structured Outputs Mutipe Output Dependency? (MOD) New research area CS 478 Backpropagation 35

36 Locaist vs. Distributed Representations Is Memory Locaist ( grandmother ce ) or distributed Output Nodes One node for each cass (cassification) One or more graded nodes (cassification or regression) Distributed representation Input Nodes Normaize rea and ordered inputs Nomina Inputs - Same options as above for output nodes Hidden nodes - Can potentiay extract rues if ocaist representations are discovered. Difficut to pinpoint and interpret distributed representations. CS 478 Backpropagation 36

37 Appication Exampe - NetTak One of first appication attempts Train a neura network to read Engish aoud Input Layer - Locaist representation of etters and punctuation Output ayer - Distributed representation of phonemes 120 hidden units: 98% correct pronunciation Note steady progression from simpe to more compex sounds CS 478 Backpropagation 37

38 Regression with MLP/BP Hidden nodes sti use a standard non-inear activation function (e.g. sigmoid) Repace output node with a inear activation (i.e. identity which just passed the net vaue through) which more naturay supports unconstrained regression Since f '(net) is 1 for the inear activation, the output error is just (target output) The hidden ayer errors have the standard f '(net) CS 478 Backpropagation 38

39 Rectified Linear Units f(x) = Max(0,x) More efficient gradient propagation, derivative is 0 or constant, just fod into earning rate More efficient computation: Ony comparison, addition and mutipication. Leaky ReLU f(x) = x if x > 0 ese ax, where 0 a <= 1, so that derivate is not 0 and can do some earning for net < 0 (does not die ). Lots of other variations Sparse activation: For exampe, in a randomy initiaized networks, ony about 50% of hidden units are activated (having a non-zero output) CS 678 Deep Learning 39

40 Softmax and Cross-Entropy Sum-squared error (L2) oss gradient seeks the maximum ikeihood hypothesis under the assumption that the training data can be modeed by Normay distributed noise added to the target function vaue. Fine for regression but ess natura for cassification. For cassification probems it is advantageous and increasingy popuar to use the softmax activation function, just at the output ayer, with the cross-entropy oss function. Softmax (softens) 1 of n targets to mimic a probabiity vector for each output. f (net j ) = Cross entropy seeks to to find the maximum ikeihood hypotheses under the assumption that the observed (1 of n) Booean outputs is a probabiistic function of the input instance. Maximizing ikeihood is cast as the equivaent minimizing of the negative og ikeihood. Loss CrossEntopy = t i n(z i ) With new oss and activation functions, we must recacuate the gradient equation. Gradient/Error on the output is just (t-z), no f '(net)! The exponent of softmax is unraveed by the n of cross entropy. Heps avoid gradient saturation. Common with deep networks, with ReLU activations for the hidden nodes e net j n e net i i=1 n i=1 CS 478 Backpropagation 40

41 Batch Update With On-ine (stochastic) update we update weights after every pattern With Batch update we accumuate the changes for each weight, but do not update them unti the end of each epoch Batch update gives a correct direction of the gradient for the entire data set, whie on-ine coud do some weight updates in directions quite different from the average gradient of the entire data set Based on noisy instances and aso just that specific instances wi not represent the average gradient Proper approach? - Conference experience Most (incuding us) assumed batch more appropriate, but batch/onine a non-critica decision with simiar resuts We tried to speed up earning through "batch paraeism" CS 478 Backpropagation 41

42 On-Line vs. Batch Wison, D. R. and Martinez, T. R., The Genera Inefficiency of Batch Training for Gradient Descent Learning, Neura Networks, vo. 16, no. 10, pp , 2003 Many peope sti not aware of this issue Changing Misconception regarding Fairness in testing batch vs. on-ine with the same earning rate BP aready sensitive to LR - why? Both approaches need to make a sma step in the cacuated gradient direction (about the same magnitude) With batch need a "smaer" LR since weight changes accumuate (aternativey divide by TS ) To be "fair", on-ine shoud have a comparabe LR?? Initiay tested on reativey sma data sets On-ine update approximatey foows the curve of the gradient as the epoch progresses With appropriate earning rate batch gives correct resut, just ess efficient, since you have to compute the entire training set for each sma weight update, whie on-ine wi have done TS updates CS 478 Backpropagation 42

43 Point of evauation Direction of gradient True underying gradient CS 478 Backpropagation 43

44 CS 478 Backpropagation 44

45 CS 478 Backpropagation 45

46 CS 478 Backpropagation 46

47 CS 478 Backpropagation 47

48 CS 478 Backpropagation , , , % 96.13% 95.39% 84.13% 96.49% 96.49% 95.76% 95.20% 23.25% 96.49% 96.68% 96.13% 90.77% 96.68% 96.49% 96.49% 96.31% Learning Rate Batch Size Max Word Accuracy Training Epochs Semi-Batch on Digits

49 On-Line vs. Batch Issues Some say just use on-ine LR but divide by n (training set size) to get the same feasibe LR for both (non-accumuated), but on-ine sti does n times as many updates per epoch as batch and is thus much faster True Gradient - We just have the gradient of the training set anyways which is an approximation to the true gradient and true minima Momentum and true gradient - same issue with other enhancements such as adaptive LR, etc. Training sets are getting arger - makes discrepancy worse since we woud do batch update reativey ess often Large training sets great for earning and avoiding overfit - best case scenario is huge/infinite set where never have to repeat - just 1 partia epoch and just finish when earning stabiizes batch in this case? Sti difficut to convince some peope CS 478 Backpropagation 49

50 Adaptive Learning Rate/Momentum Momentum is a simpe speed-up modification Δw(t+1) = Cδ x i + αδw(t) Are we true gradient descent when using this? Note it is kind of a mini-batch foowing the oca avg gradient Weight update maintains momentum in the direction it has been going Faster in fats Coud eap past minima (good or bad), but not a big issue in practice Significant speed-up, common vaue α.9 Effectivey increases earning rate in areas where the gradient is consistenty the same sign. Adaptive Learning rate methods Start LR sma As ong as weight change is in the same direction, increase a bit (e.g. scaar mutipy > 1, etc.) If weight change changes directions (i.e. sign change) reset LR to sma, coud aso backtrack for that step, or CS 478 Backpropagation 50

51 Speed up variations of SGD Use mini-batch rather than singe instance for better gradient estimate Sometimes hepfu if using GD variation more sensitive to bad gradient, and aso for some parae impementations. Adaptive earning rate approaches (and other speed-ups) are often used for deep earning since there are so many training updates Standard Momentum Note the these approaches aready do an averaging of grading aso making mini-batch ess critica Nesterov Momentum Cacuate point you woud go to if using norma momentum. Then, compute gradient at that point. Do norma update using that gradient and momentum. Rprop Resiient BP, if gradient sign inverts, decrease it s individua LR, ese increase it common goa is faster in the fats, variants that backtrack a step, etc. Adagrad Scae LRs inversey proportiona to sqrt(sum( historica vaues)) LRs with smaer derivatives are decreased ess RMSprop Adagrad but uses exponentiay weighted moving average, oder updates basicay forgotten Adam (Adaptive moments) Momentum terms on both gradient and squared gradient (1 st and 2 nd moments) update based on both CS 678 Deep Learning 51

52 Learning Variations Different activation functions - need ony be differentiabe Different objective functions Cross-Entropy Cassification Based Learning Higher Order Agorithms - 2nd derivatives (Hessian Matrix) Quickprop Conjugate Gradient Newton Methods Constructive Networks Cascade Correation DMP (Dynamic Muti-ayer Perceptrons) CS 478 Backpropagation 52

53 Cassification Based (CB) Learning Target Actua BP Error CB Error 1.6.4*f '(net) *f '(net) *f '(net) 0 CS 478 Backpropagation 53

54 Cassification Based Errors Target Actua BP Error CB Error 1.6.4*f '(net) *f '(net) *f '(net) 0 CS 478 Backpropagation 54

55 Resuts Standard BP: 97.8% Sampe Output: CS 478 Backpropagation 55

56 Resuts Cassification Based Training: 99.1% Sampe Output: CS 478 Backpropagation 56

57 Anaysis Correct Incorrect # Sampes Top Output Network outputs on test set after standard backpropagation training. CS 478 Backpropagation 57

58 Anaysis Correct Incorrect # Sampes Top Output Network outputs on test set after CB training. CS 478 Backpropagation 58

59 Cassification Based Modes CB1: Ony backpropagates error on miscassified training patterns CB2: Adds a confidence margin, µ, that is increased gobay as training progresses CB3: Learns a confidence C i for each training pattern i as training progresses Patterns often miscassified have ow confidence Patterns consistenty cassified correcty gain confidence Best overa resuts and robustness CS 478 Backpropagation 59

60 Recurrent Networks Output t one step time deay one step time deay Hidden/Context Nodes Input t Some probems happen over time - Speech recognition, stock forecasting, target tracking, etc. Recurrent networks can store state (memory) which ets them earn to output based on both current and past inputs Learning agorithms are somewhat more compex and ess consistent than norma backpropagation Aternativey, can use a arger snapshot of features over time with standard backpropagation earning and execution (e.g. NetTak) CS 478 Backpropagation 60

61 Backpropagation Summary Exceent Empirica resuts Scaing The peasant surprise Loca minima very rare as probem and network compexity increase Most common neura network approach Many other different styes of neura networks (RBF, Hopfied, etc.) User defined parameters usuay handed by mutipe experiments Many variants Regression Typicay Linear output nodes, norma hidden nodes Adaptive Parameters, Ontogenic (growing and pruning) earning agorithms Many different earning agorithm approaches Higher order gradient descent (Newton, Conjugate Gradient, etc.) Recurrent networks Deep networks! Sti an active research area CS 478 Backpropagation 61