Robust Boosting and its Relation to Bagging

Transcription

1 Robust Boostng and ts Relaton to Baggng Saharon Rosset IBM T.J. Watson Research Center P. O. Box 218 Yorktown Heghts, NY ABSTRACT Several authors have suggested vewng boostng as a gradent descent search for a good ft n functon space. At each teraton observatons are re-weghted usng the gradent of the underlyng loss functon. We present an approach of weght decay for observaton weghts whch s equvalent to robustfyng the underlyng loss functon. At the extreme end of decay ths approach converges to Baggng, whch can be vewed as boostng wth a lnear underlyng loss functon. We llustrate the practcal usefulness of weght decay for mprovng predcton performance and present an equvalence between one form of weght decay and Huberzng a statstcal method for makng loss functons more robust. Categores and Subject Descrptors G.3 [Mathematcs of Computng]: Probablty and Statstcs; I.5.2 [Pattern Recognton]: Desgn Methodology General Terms Algorthms Keywords Boostng, Baggng, Robust Fttng 1. INTRODUCTION Boostng [9, 8] and Baggng [3] are two approaches to combnng weak models n order to buld predcton models that are sgnfcantly better. Much has been wrtten about the emprcal success of these approaches n creatng predcton models n actual modelng tasks [4, 1]. The theoretcal dscussons of these algorthms [4, 11, 12, 5, 16] have vewed them from varous perspectves. The general theoretcal and practcal consensus, however, s that the weak learners for boostng should be really weak, whle the weak learners for baggng should actually be strong. In tree termnology, one should use small trees when boostng and bg Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, to republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. KDD 05, August 21 24, 2005, Chcago, Illnos, USA. Copyrght 2005 ACM X/05/ $5.00. trees for baggng. In ntutve bas-varance terms, we can say that baggng s manly a varance reducton (or stablzaton) operaton, whle boostng, n the way t flexbly combnes models, s also a bas reducton operaton,.e., t adds flexblty to the representaton beyond that of a sngle learner. In ths paper we present a vew of boostng and baggng whch allows us to connect them n a natural way, and n fact vew baggng as a form of boostng. Ths vew s nterestng because t facltates creaton of famles of ntermedate algorthms, whch offer a range of degrees of bas reducton between boostng and baggng. Our experments ndcate that, whle baggng s always sgnfcantly nferor to boostng n terms of predctve performance, n some cases ntermedate approaches can outperform standard boostng. Our exposton concentrates on 2-class classfcaton, ths beng the most common applcaton for both boostng and baggng, but the results mostly carry through to other learnng domans where boostng and baggng have been used, such as mult-class classfcaton, regresson [10] and densty estmaton [17]. 2. BOOSTING, BAGGING AND A CONNECTION A vew has emerged n the last few years of boostng as a gradent-based search for a good model n a large mplct feature space [16, 10]. More specfcally, the components of a 2-class classfcaton boostng algorthm are: A data sample {x, y } n =1, wth x R p and y { 1, +1} A loss functon C(y, f). The two most commonly used are the exponental loss of AdaBoost [9] and the logstc log-lkelhood of LogtBoost [11]. We wll assume throughout that C(y, f) s a monotone decreasng and convex functon of the margn m = yf, and so we wll sometmes wrte t as C(m). A dctonary H of weak learners, where each h H s a classfcaton model: h : R p { 1, +1}. The dctonary most commonly used n classfcaton boostng s classfcaton trees,.e., H s the set of all trees wth up to k splts. Gven these components, a boostng algorthm ncrementally bulds a good lnear combnaton of the dctonary functons: F (x) = h H β h h(x) 249

2 Where good s defned as makng the emprcal loss small: C(y F (x )) The actual ncremental algorthm s an exact or approxmate coordnate descent algorthm. At teraton t we have the current ft F t, and we look for the weak learner h t whch maxmzes the frst order decrease n the loss,.e., h t maxmzes C(y, F (x )) F =Ft β h or equvalently and more clearly t maxmzes C(y, F (x )) F =Ft h(x ) F (x ) whch n the case of two-class classfcaton s easly shown to be equvalent to mnmzng w I{y h(x )} (1) Where w = C(y,F (x )) F (x ) F =Ft. So we seek to fnd a weak learner whch mnmzes weghted error rate, wth the weghts beng the gradent of the loss. If we use the exponental loss: C(y, f) = exp( yf) (2) then t can be shown (e.g. [13]) that (1) s the exact classfcaton task whch AdaBoost [9], the orgnal and most famous boostng algorthm, solves for fndng the next weak learner. In ther orgnal AdaBoost mplementaton [8], Freund and Schapre suggested solvng (1) on a new tranng data set of sze n at each teraton, by samplng from the tranng dataset wth return wth probabltes proportonal to w. Ths facltates the use of methods for solvng non-weghted classfcaton problems for approxmately solvng (1). We wll term ths approach the samplng boostng algorthm. [10] has argued that samplng actually mproves the performance of boostng algorthms, by addng much needed randomness (hs approach s to solve the weghted verson of (1) on a sub-sample, but the basc motvaton apples to samplng boostng as well). Here s a formal descrpton of a samplng boostng algorthm, gven the nputs descrbed above: Algorthm 1. Generc gradent-based samplng boostng algorthm 1. Set β 0 = 0 (dmenson of β s H ). 2. For t = 1 : T, Comments: (a) Let F = β T t 1h(x ), = 1,..., n (the current ft, where h(x) s the vector of weak learner values at x). (b) Set w = C(y,F ) F, = 1,..., n. (c) Draw a sample {x, y } n =1 of sze n by re-samplng wth return from {x, y } n =1 wth probabltes proportonal to w (d) Identfy j t = arg mn j I{y h j(x )}. (e) Set β t,jt = β t 1,jt + ɛ and β t,k = β t 1,k, k j t. 1. Implementaton detals nclude the determnaton of T (or other stoppng crteron) and the approach for fndng the mnmum n step 2(d). 2. We have fxed the step sze to ɛ at each teraton (step 2(e)). Whle AdaBoost uses a lne search to determne step sze, t can be argued that a fxed (usually small ) ɛ step s theoretcally preferable (see [10, 18] for detals). 3. An mportant ssue n desgnng boostng algorthms s the selecton of the loss functon C(, ). Extreme loss functons, such as the exponental loss of AdaBoost (2), are not robust aganst outlers and msspecfed data, as they assgn overwhelmng weght to observatons whch have the smallest margns. [11] have thus suggested replacng the exponental loss wth the logstc log lkelhood loss: C(y, f) = log(1 + exp( yf)) (3) but n many practcal stuatons, n partcular when the two classes n the tranng data are separable n sp(h), ths loss can also be non-robust. See Secton The algorthm as descrbed here s not affected by postve affne transformatons of the loss functon,.e., runnng Algorthm 1 usng a loss functon C(m) s exactly the same as usng C (m) = ac(m) + b as long as a > 0. Baggng for classfcaton [3] s a dfferent model combnng approach. It searches, n each teraton, for the member of the dctonary H whch best classfes a bootstrap sample of the orgnal tranng data, and then averages the dscovered models to get a fnal bagged model. So, n fact, we can say that: Proposton 1. Baggng mplements Algorthm 1, usng a lnear loss, C(y, f) = yf (or any postve affne transformaton of t) Proof. From the defnton of the loss we get: F t,, yf (x) F =Ft 1 F (x ) So no matter what our current model s, all the gradents are always equal, hence the weghts wll be equal f we apply a samplng boostng teraton usng the lnear loss. In the case that all weghts are equal, the boostng samplng procedure descrbed above reduces to bootstrap samplng. Hence the baggng algorthm s a samplng boostng algorthm wth a lnear loss. Thus, we can consder Baggng as a boostng algorthm utlzng a very robust (lnear) loss. Ths loss s so robust that t requres no message passng between teratons through re-weghtng, snce the gradent of the loss does not depend on the current margns. 2.1 Robustness, convexty and boostng The gradent of a lnear loss does not emphasze lowmargn data ponts over hgh-margn ( well predcted ) ones. In fact, t s the most robust convex loss possble, n the followng sense: 250

3 Proposton 2. Any loss whch s a dfferentable, convex and decreasng functon of the margn has the property: m 1 < m 2 C (m 1 ) C (m 2 ) And a lnear loss s the only one whch attans equalty m 1, m 2. Proof. Immedate from convexty and monotoncty. [16] have used generalzaton error bounds to argue that a good loss for boostng would be even more robust than the lnear loss, and consequently non-convex. In partcular, they argue that both hgh-margn and low-margn observatons should have low weght, leadng to a sgmod-shaped loss functon. Non-convex loss functons present a sgnfcant computatonal challenge, whch [16] have solved for small dctonary examples. Although the dea of such outler tolerant loss functons s appealng, we lmt our dscusson to convex loss functons, whch facltate the use of standard fttng methodology, n partcular boostng. Our vew of baggng as boostng wth lnear loss allows us to nterpret the smlarty and dfference between the algorthms by lookng at the loss functons they are boostng. The lnear loss of baggng mples t s not emphaszng the badly predcted observatons, but rather treats all data equally. Thus t s more robust aganst outlers and more stable, but less adaptable to the data than boostng wth an exponental or logstc loss. 3. BOOSTING WITH WEIGHT DECAY The vew of baggng as a boostng algorthm, opens the door to creatng boostng-baggng hybrds, by robustfyng the loss functons used for boostng. These hybrds may combne the advantages of boostng and baggng to gve us new and useful algorthms. There are two ways to go about creatng these ntermedate algorthms: Defne a seres of loss functons startng from a boostng loss the exponental or logstc and convergng to the lnear loss of baggng. Implctly defne the ntermedate loss functons by decayng the weghts w gven by the boostng algorthm usng a boostng loss. The loss mpled wll be the one whose gradent corresponds to the decayed weghts. The two approaches are obvously equvalent through a dfferentaton or ntegraton operaton. We wll adopt the weght decay approach, but wll dscuss the loss functon mplcatons of the dfferent decay schemes. 3.1 Weght decay functons We would lke to change the loss C(, ) to be more robust, by frst decayng the (gradent) weghts w, then consderng the mplct effect of the decay on the loss. In general, we assume that we have a decay functon v(p, w) whch depends on a decay parameter p [0, 1] and the observaton weght w 0. We requre: v(1, w) = w,.e., no decay v(0, w) = 1,.e., no weghtng, whch mplctly assumes a lnear loss when consdered as a boostng algorthm and thus corresponds to Baggng Monotoncty: w 1 < w 2 v(p, w 1 ) < v(p, w 2 ) p. Contnuty n both p and w. And once we specfy a decay parameter p, the problem we solve n each boostng teraton, nstead of (1), s to fnd the dctonary functon h to mnmze: v(p, w )I{y h(x )} (4) whch we solve approxmately as a non-weghted problem, usng the samplng boostng approach of Algorthm Wndsorsed weghts and Huberzed loss For clarty and brevty, we concentrate n ths paper on one decay functon the boundng or Wndsorsng operator: v(p, w) = { 1 f w > p 1 p 1 p otherwse w p Snce w s the absolute gradent of the loss, boundng w means that we are huberzng the loss functon,.e., contnung t lnearly beyond some pont (ths operaton was suggested, n the context of squared error loss for regresson, by [14]). The loss functon correspondng to the decayed weghts s thus: C (p) (m) = { C(m) p C(m (p)) p m m (p) 1 p m > m (p) otherwse where m (p) s such that C (m (p)) = p (unque because the loss s convex and monotone 1 p decreasng) p=1 p=0.4 p=0.1 p= Fgure 1: Huberzed logstc loss, for dfferent values of p (as a functon of the margn). As we have mentoned, for the purpose of the samplng boostng algorthm, only the relatve szes of the weghts are mportant, and thus multplcaton of the loss functon by a constant of 1/p whch we have done to get C (p) from C and acheve the desred propertes of v(p, w) does not affect the algorthm for fxed p. Fgure 1 llustrates the effect of boundng on the logstc log lkelhood loss functon, for several values of p (for presentaton, ths plot uses the non-scaled verson of C,.e., (6) multpled by p). There are many other nterestng decay functons, such as the power transformaton: v(p, w) = w p (5) (6) 251

4 Ths transformaton s attractve because t does not ental the arbtrary threshold determnaton, but rather decays all weghts, wth the bgger ones decayed more. However, t s less nterpretable n terms of ts effect on the underlyng loss. For the exponental loss C(m) = exp( m), the power transformaton does not change the loss, snce [exp( m)] p = exp( mp). So the effect of ths transformaton s smply to slow the learnng rate (exactly equvalent to decreasng ɛ n Algorthm 1). 4. PROPERTIES OF HUBERIZED BOOSTING LOSS FUNCTIONS Our suggested huberzng transformaton n (6) to the orgnal loss functon (n our case, the exponental loss (2) or the logstc loss (3)) gves a modfed loss whch s lnear for small margns, then as the margn ncreases starts behavng lke the orgnal loss. We can therefore nterpret boostng wth the huberzed loss functon as baggng for a whle, untl the margns become bg enough and reach the nonhuberzed regon. If we boost long enough wth ths loss, we wll have most of the data margns n the non-huberzed regon, except for extreme outlers, and the resultng weghtng scheme wll revert to the standard boostng weghts, except for extreme outlers. We can thus consder boostng wth the huberzed loss to be more robust n the followng sense: By huberzng the loss functon, we are allowng the boostng algorthm to assgn relatvely small weghts to outlers and dffcult examples for a number of teratons, untl most observatons have become well explaned by attanng large margns Ths robustness argument s related to that of [11] and others for boostng wth the logstc loss (3) rather than the exponental (2), snce the logstc loss s approxmately lnear for negatve margns (as ts second dervatve vanshes when the margn goes to ). However, the logstc loss s very smlar to the exponental for non-negatve margns, as shown n [18]. Huberzng, on the other hand, allows for a flexble and specfc defnton of a lnear regon whch can be adapted to the modelng task at hand. It s nterestng to note that two desrable propertes of boostng loss functons are mantaned for ther huberzed versons: The boostng property Ths property states that f the hypothess space of weak learners allows weak learnng that s, weghted error rate of the weak learners of at most 1/2 λ for each teraton then the tranng margns wll ncrease, and generalzaton error wll go to 0 for large enough tranng samples (see [20] for more detals). Duffy and Helmbold ([7], Theorems 2-4) gve suffcent condtons for ths property to hold, whch apply to the exponental and logstc loss functons. These condtons nclude strct convexty, and so do not drectly apply to the huberzed versons. However closer nspecton of ther condtons exposes that we only need to re-state ther Theorem 2, whch states that all the (non-normalzed) margns are guaranteed to move beyond a fxed pont U wthn O(n 2 /λ 2 ) teratons, and never cross t back. We re-prove ths result for Huberzed loss functons, and take U = m to be the Huberzng pont. Once all the margns have moved beyond m, the loss functon becomes the logstc or exponental, whch are strctly convex, and Duffy and Helmbold s results prove the boostng property 1. Theorem 3. Assume the weak learnablty property for Algorthm 1,.e., w I{h t (x ) y } 1 λ, t (7) w 2 Then for the huberzed verson (6) of ether the exponental loss or the logstc loss, we get that after at most T = 4n2 C (p) (0) C (p) (m (p))λ 2 teratons we acheve total loss no bgger than C (p) (m (p)). That s: C (p) (y, F t (x )) C (p) (m (p)), t T and, consequently, all margns are guaranteed to be no bgger than m (p) from that teraton on. Proof outlne. Combnng the defnton of w wth (7) we get at each teraton: C(y, F (x )) F =Ft > 2λ β ht w (8) If our overall loss s bgger than C (p) (m (p)) then we can show that: w > C(p) (m (p)) 2 (for the exponental loss ths s a drect result of the property: C(p) (m) m = C(p) (m) and for the logstc loss we also use Proposton 1 of [18]). Next, we bound the second dervatve of the change n loss: 2 C (p) (y, F (x )) β 2 h t F =Ft 2 C (p) (m ) m 2 F =Ft (9) 2nC (p) (m (p)) (10) because the loss s convex and ts second dervatve s provably bounded by 2C (p) (m (p)). Takng these three facts (8,9,10) combned we get that at each boostng teraton f we take a step of sze λ we gan 2n an mprovement n the loss of at least: C (p) (m (p)) λ 2 4n as long as the total loss exceeds C (p) (m (p)). The actual step whch a lne-search boostng algorthm takes can be 1 Note that the generalzaton error property s proven for lne search boostng algorthms, not the ɛ-boostng approach we employ n our mplementaton below. [7] s results therefore mply that our suggested loss functons would also enjoy ths property, f embedded nto lne-search boostng mplementatons. 252

5 bgger than λ, but the mprovement n loss s guaranteed 2n to be at least that attaned by takng ths fxed step sze. Now, we observe that our ntal loss s nc (p) (0) and thus wthn ths number of teratons: T = 4n2 C (p) (0) C (p) (m (p))λ 2 we are guaranteed to have a non-postve loss f we stll have loss bgger than C (p) (m (p)). Ths obvously gves a contradcton and we conclude that the total loss after T teratons gves the desred result: C (p) (y, F T (x )) C (p) (m (p)) Snce the overall loss always decreases n every teraton n lne-search boostng, t can never exceed C (p) (m (p)) n subsequent teratons. Snce the loss functon s non-negatve decreasng, no margn can ever be smaller than m (p) n subsequent teratons. The margn maxmzng property [19] gve suffcent condtons for regularzed loss functons to be l p-margn maxmzng. Ths suffcent condton holds for the logstc and exponental loss functons, and also for ther huberzed versons, snce t only depends on the loss functon s behavor as the non-normalzed margn converges to. [18] explan how ths property extends approxmately to boostng. In essence, t mples that the robust boostng algorthm s seekng to maxmze the l 1 margn of the data examples, and under qute general condtons wll succeed n dong so. 5. EXPERIMENTS We now dscuss brefly some experments to examne the usefulness of weght decay and the stuatons n whch t may be benefcal. We use three datasets: the Spam and Waveform datasets, avalable from the UCI repostory [2]; and the Dgts handwrtten dgts recognton dataset, dscussed n [15]. These were chosen snce they are reasonably large and represent very dfferent problem domans. Snce we have lmted the dscusson here to 2-class models, we selected only two classes from the mult-class datasets: waveforms 1 and 2 from Waveform and the dgts 2 and 3 from Dgts (these were selected to make the problem as challengng as possble). The resultng 2-class data-sets are reasonably large rangng n sze between about 2000 and over In all three cases we used only 25% of the data for tranng and 75% for evaluaton, as our man goal s not to excel on the learnng task, but rather to make t dffcult and expose the dfferences between the models whch the dfferent algorthms buld. Our experments conssted of runnng Algorthm 1 usng varous loss functons, all obtaned by decayng the observaton weghts gven by the exponental loss functon (2). We used Wndsorsng decay as n (5) and thus the decayed versons correspond to huberzed versons of (2). In all our experments, baggng performed sgnfcantly worse than all versons of boostng, whch s consstent wth observatons made by varous researchers, that well-mplemented boostng algorthms almost nvarably domnate baggng (see for example Breman s own experments n [4]). It should be clear, however, that ths fact does not contradct the vew that baggng has some desrable propertes, n partcular a greater stablzng effect than boostng. We present n Fgure 2 the results of runnng Algorthm 1 wth p = 1 (.e., usng the non-decayed loss functon (2)), and wth p = (whch corresponds to huberzng the loss, as n (6), at around m = 9) on the three data-sets. We use two settngs for the weak learner : 10-node and 100-node trees. The learnng rate parameter ɛ s fxed at 0.1. The results n Fgure 2 represent averages over 20 random tran-test splts, wth estmated 2-standard devatons confdence bounds. These plots expose a few nterestng observatons: 1. Weght decay leads to a slower learnng rate. From an ntutve perspectve ths s to be expected, as a more robust loss corresponds to less aggressve learnng, by puttng less emphass on the hardest cases. 2. Weght decay s more useful when the weak learners are not weak, but rather strong, lke a 100-node tree. Ths s partcularly evdent for the Spam dataset, where the performance of the non-decayed exponental boostng deterorates sgnfcantly when we move from 10-node to 100-node learners, whle that of the decayed verson actually mproves sgnfcantly. Ths phenomenon s also as we expect, gven the more robust Huberzed loss mpled by the decay. 3. There s no consstent wnner between the non-decayed and the decayed versons. For the Spam and Waveform datasets t seems that f we choose the best settngs, we would choose the non-decayed loss wth small trees. For the Dgts data, the decayed loss seems to produce consstently better predcton models. Note that n our examples we dd not have bg robustness ssues as they pertan to extreme or hghly prevalent outlers n the predctors or the response. Rather, we examne the bas-varance tradeoff n employng more robust loss functons n dealng wth uncorrupted real-lfe datasets, and the dffcult cases they often contan. It s lkely that n extreme stuatons of many and/or bg outlers the advantage of usng robust loss functons would be more pronounced. 6. DISCUSSION The gradent descent vew of boostng allows us to desgn boostng algorthms for a varety of problems, and choose the loss functons whch we deem most approprate. [18] show that gradent boostng approxmately follows a path of l 1 -regularzed solutons to the chosen loss functon. Thus selecton of an approprate loss s a crtcal ssue n buldng useful algorthms. In ths paper we have shown that the gradent boostng paradgm covers baggng as well, and used t as ratonale for consderng new famles of loss functons hybrds of standard boostng loss functons and the baggng lnear loss functon as canddates for gradent-based boostng. The results seem promsng. There are some natural extensons to ths concept, whch may be even more promsng, n partcular the dea that the loss functon does not have to reman fxed throughout the boostng teratons. Thus, we can desgn dynamc loss functons whch change as the boostng teratons proceed, ether as a functon of the performance of the model or ndependently of t. It seems that 253

6 0.065 Spam 10 node trees Spam 100 node trees Waveform 10 node trees Waveform 100 node trees Dgts 10 node trees Dgts 100 node trees Fgure 2: Results of runnng samplng boostng wth p = 1 (sold) and p = (dashed) on the three datasets. wth 10-node and 100-node trees as the weak learners. The x-axs s the number of teratons, and the y-axs s the mean test-set error over 20 random tranng-test assgnments. The dotted lnes are 2-sd confdence bounds. 254

7 there are a lot of nterestng theoretcal and practcal questons that should come nto consderaton when we desgn such an algorthm, such as: Should the loss functon become more or less robust as the boostng teratons proceed? Should the loss functon become more or less robust f there are problematc data ponts? In ths context, t s nterestng to note some prevous work on boundng the boostng weghts to acheve more robust performance by [6]. The man dfference from our approach s that they operate on the re-normalzed AdaBoost weghts. Ther approach thus lacks the gradent descent nterpretaton n a new loss, snce AdaBoost s re-normalzaton of the weghts s equvalent to re-scalng the underlyng loss. Thus, [6] s approach amounts to Huberzng the exponental loss at a dfferent pont n each teraton. Ther algorthm also lacks a guaranteed boostng property lke the one we proved n Secton 4. However t would be nterestng to compare the emprcal mert of ther approach to ours and perhaps draw conclusons wth regard to usng adaptve robust loss functons. 7. ACKNOWLEDGMENTS Jerry Fredman, Trevor Haste and J Zhu contrbuted to ths paper through useful dscussons and advce. 8. REFERENCES [1] E. Bauer and R. Kohav. An emprcal comparson of votng classfcaton algorthms: baggng, boostng and varants. Machne Learnng, 36(1/2): , [2] C. Blake and C. Merz. Repostory of machne learnng databases. [ mlearn/mlrepostory.html]. Irvne, CA: Unversty of Calforna, Department of Informaton and Computer Scence., [3] L. Breman. Baggng predctors. Machne Learnng, 24(2): , [4] L. Breman. Arcng classfers. Annals of Statstcs, 26(3):801:849, [5] P. Buhlmann and B. Yu. Analyzng baggng. Annals of Statstcs, [6] C. Domngo and O. Watanabe. Madaboost: A modfcaton of adaboost. In 13th Annual Conference on Comp. Learnng Theory, [7] N. Duffy and D. Helmbold. Potental boosters? In Advance n Neural Informaton Processng, [8] Y. Freund and R. Scahpre. Experments wth a new boostng algorthm. In Internatonal Conference on Machne Learnng, [9] Y. Freund and R. Schapre. A decson-theoretc generalzaton of on-lne learnng and an applcaton to boostng. In European Conference on Computatonal Learnng Theory, pages 23 37, [10] J. Fredman. Greedy functon approxmaton: A gradent boostng machne. Annals of Statstcs, 29(5), [11] J. Fredman, T. Haste, and R. Tbshran. Addtve logstc regresson: a statstcal vew of boostng. Annals of Statstcs, 28: , [12] J. H. Fredman and P. Hall. On baggng and nonlnear estmaton. preprnt, [13] T. Haste, T. Tbshran, and J. Fredman. Elements of Statstcal Learnng. Sprnger-Verlag, New York, [14] P. Huber. Robust estmaton of a locaton parameter. Annals of Mathematcal Statstcs, 35(1):73:101, [15] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Handwrtten dgt recognton wth a back-propagaton network. In Advances n Neural Informaton Processng Systems 2, [16] L. Mason, J. Baxter, P. Bartlett, and M. Frean. Boostng algorthms as gradent descent. In Neural Informaton Processng Systems, volume 12, [17] S. Rosset and E. Segal. Boostng densty estmaton. In Advances n Neural Informaton Processng Systems 15, [18] S. Rosset, J. Zhu, and T. Haste. Boostng as a regularzed path to a maxmum margn classfer. Journal of Machne Learnng Research, 5, [19] S. Rosset, J. Zhu, and T. Haste. Margn maxmzng loss functons. In Advances n Neural Informaton Processng Systems 16, [20] R. E. Schapre, Y. Freund, P. Bartlett, and W. Lee. Boostng the margn: a new explanaton for the effectveness of votng methods. Annals of Statstcs, 26: ,