COSC 6385 Computer Architecture. Performance Measurement

Size: px

Start display at page:

Download "COSC 6385 Computer Architecture. Performance Measurement"

Harvey Clarke
6 years ago
Views:

1 COSC 6385 Computer Archtecture Performace Measuremet Edgar Gabrel Sprg 204 Measurg performace (I) Respose tme: how log does t take to execute a certa applcato/a certa amout of work Gve two platforms X ad Y, X s tmes faster tha Y for a certa applcato f Tme Tme Y X Performace of X s tmes hgher tha performace of Y f Tme Tme Y X Perf Perf Y X Perf Perf X Y () (2)

Measurg performace (II) Tmg how log a applcato takes Wall clock tme/elapsed tme: tme to complete a task as see by the user. Mght clude operatg system overhead or potetally terferg other applcatos.

2 Measurg performace (II) Tmg how log a applcato takes Wall clock tme/elapsed tme: tme to complete a task as see by the user. Mght clude operatg system overhead or potetally terferg other applcatos. CPU tme: does ot clude tme slces troduced by exteral sources (e.g. rug other applcatos). CPU tme ca be further dvded to User CPU tme: CPU tme spet the program System CPU tme: CPU tme spet the OS performg tasks requested by the program. Measurg performace E.g. usg the UNIX tme commad Elapsed tme User CPU tme System CPU tme 2

3 Speedup overall Amdahl s Law Descrbes the performace gas by acg oe part of the overall system (code, computer) Tmeorg Perf Speedup (3) Tme Perf org Amdahl s Law depeds o two factors: Fracto of the executo tme affected by acemet The mprovemet gaed by the acemet for ths fracto Tme Tme org (( Fracto Fracto ) Speedup ) (4) Speedup overall Tme Tme org ( Fracto Fracto ) Speedup (5) 6 Speedup Amdahl s Law (III) overall Fracto ( Fracto ) Speedup Fracto aced: 20% Fracto aced: 40% Fracto aced: 60% Fracto aced: 80% Speedup aced 3

4 Speedup overall Amdahl s Law (IV) Speedup accordg to Amdahl's Law Speedup aced: 2 Speedup aced: 4 Speedup aced: Fracto aced Amdahl s Law - example Assume a ew web-server wth a CPU beg 0 tmes faster o computato tha the prevous web-server. I/O performace s ot mproved compared to the old mache. The web-server speds 40% of ts tme computato ad 60% I/O. How much faster s the ew mache overall? Fracto 0.4 Speedup 0 usg formula (5) Speedup overall ( Fracto Fracto ) Speedup 0.4 ( 0.4)

5 Amdahl s Law example (II) Example: Cosder a graphcs card 50% of ts total executo tme s spet floatg pot operatos 20% of ts total executo tme s spet floatg pot square root operatos (FPSQR). Opto : mprove the FPSQR operato by a factor of 0. Opto 2: mprove all floatg pot operatos by a factor of.6 Speedup FPSQR 0.2 ( 0.2) ( ) 0 Speedup FP 0.5 ( 0.5) ( ) Opto 2 slghtly faster CPU Performace Equato Mcro-processors are based o a clock rug at a costat rate Clock cycle tme: CC t legth of the dscrete tme evet s Equvalet measure: Rate CPU r CCtme Expressed MHz, GHz CPU tme of a program ca the be expressed as or CPU CPU tme tme o cycles o CPU cycles r CC tme (6) (7) 5

6 CPU Performace equato (II) CPI: Average umber of clock cycles per structo IC: umber of structos o CPI IC cycles Sce the CPI s ofte kow (average), the CPU tme s (8) CPU IC CPI tme CC tme Expadg formula (6) leads to CPU tme structos ocycles program structo tme o cycles (9) (0) CPU performace equato (III) Accordg to (7) CPU performace s depedg o Clock cycle tme Hardware techology CPI Orgazato ad structo set archtecture Istructo cout ISA ad compler techology Note: o the last slde we used the average CPI over all structos occurrg a applcato Dfferet structos ca have strogly varyg CPI s o cycles CPU tme IC CPI IC CPI CC tme () (2) 6

7 CPU performace equato (IV) The average CPI for a applcato ca the be calculated as CPI IC CPI IC total IC IC total CPI (3) IC IC total : Fracto of occurrece of that structo a program Example (I) (Page 50/5 the 5 th Edto) Cosder a graphcs card, wth FP operatos (cludg FPSQR): frequecy 25%, average CPI 4.0 FPSQR operatos oly: frequecy 2%, average CPI 20 all other structos: average CPI Desg opto : decrease CPI of FPSQR to 2 Desg opto 2: decrease CPI of all FP operatos to 2.5 Usg formula (3): CPI org IC IC CPI CPI total CPI (20 2) org (4*0.25) ( *0.75) IC CPI 2 CPI (2.5*0.25) ( *0.75).625 IC total 7

8 Example (II) Slghtly modfed compared to the prevous secto: cosder a graphcs card, wth FP operatos (excludg FPSQR): frequecy 25%, average CPI 4.0 FPSQR operatos: frequecy 2%, average CPI 20 all other structos: average CPI.33 Desg opto : decrease CPI of FPSQR to 2 Desg opto 2: decrease CPI of all FP operatos to 2.5 Usg formula (3): IC CPI org CPI (4*0.25) (20*0.02) (.33*0.73) ICtotal IC CPI CPI (4*0.25) (2*0.02) (.33*0.73) ICtotal IC CPI 2 CPI (2.5*0.25) (20*0.02) (.33*0.73).9959 IC total Depedablty Module relablty measures MTTF: mea tme to falure FIT: falures tme FIT MTTF Ofte expressed as falures,000,000,000 hours MTTR: mea tme to repar MTBF: mea tme betwee falures Module avalablty: MTBF MTTF MTTR M A MTTF MTTF MTTR (4) (5) (6) 8

9 Depedablty - example Assume a dsk subsystem wth the followg compoets ad MTTFs: 0 dsks, MTTF=,000,000h SCSI cotroller, MTTF=500,000h power supply, MTTF=200,000h fa, MTTF= 200,000h SCSI cable, MTTF=,000,000h What s the MTTF of the etre system? What s the probablty, that the system fals wth a week perod? Depedablty example (II) Determe the sum of the falures tme of all compoets FIT system 0,000, , , ,000,000, ,000,000 MTTF FIT 23,000,000 43, system 500 system h 23,000,000,000,000 Probablty that the system fals wth a week perod: week P system 24*7 43,500 0, ,386% 9

10 Depedablty example (III) What happes f we add a secod power supply ad we assume, that the MTTR of a power supply s 24 hours? Assumpto: falures are ot correlated MTTF of the par of power supples s the mea tme to falure of the overall system dvded by the probablty, that the redudat ut fals before the prmary ut has bee replaced MTTF of the overall system: FIT system = /MTTF power + /MTTF power =2/MTTF power MTTF system= / FIT system =MTTF power /2 Depedablty example (III) Probablty, that ut fals wth MTTR: MTTR/ MTTF power MTTF par MTTF MTTR MTTF 2 power / power 2 2 MTTF 200,000 2MTTR ,000,000 0

11 Depedablty example (III) More geerally, f power supply has a MTTF of MTTF power_ power supply 2 has a MTTF of MTTF power_2 FIT system = /MTTF power_ + /MTTF power_2 MTTF system= / FIT system MTTF par = MTTF system /(MTTR/m(MTTF power_, MTTF power_2 )) Or f ether power_ or power_2 have bee clearly declared to be the backup ut MTTF par = MTTF system /(MTTR/MTTF power_backup ) From st quz fve years ago I order to mmze the MTTF of ther msso crtcal computers, each program o the space shuttle s executed by two computers smultaeously. Computer A s from maufacturer X ad has a MTTF of 40,000 hours, whle computer B s from maufacturer Y ad has a dfferet MTTF. The overall MTTF of the system s 4,000,000 hours. How large s the probablty that the etre system (.e. both computers) fals durg a 400 hour msso? What MTTF does the secod/backup computer have f the MTTF of the overall system s 4,000,000 hours, assumg that Computer A s falg rght at the begg of the 400 hour msso ad the MTTR s 400h (.e. the system ca oly be repared after ladg)? Soluto wll be posted o the web, but try frst o your ow!

12 Choosg the rght programs to test a system Most systems host a wde varety of dfferet applcatos Profles of certa systems gve by ther purpose/fucto Web server: hgh I/O requremets hardly ay floatg pot operatos A system used for weather forecastg smulatos Very hgh floatg pot performace requred Lots of ma memory Number of processors have to match the problem sze calculated order to delver at least real-tme results Choosg the rght programs to test a system (II) Real applcato: use the target applcato for the mache order to evaluate ts performace Best soluto f applcato avalable Modfed applcatos: real applcato has bee modfed order to measure a certa feature. E.g. remove I/O parts of a applcato order to focus o the CPU performace Applcato kerels: focus o the most tme-cosumg parts of a applcato E.g. extract the matrx-vector multply of a applcato, sce ths uses 80% of the user CPU tme. 2

13 Choosg the rght programs to test a system (III) Toy bechmarks: very small code segmets whch produce a predctable result E.g. seve of Eratosthees, qucksort Sythetc bechmarks: try to match the average frequecy of operatos ad operads for a certa program Code does ot do ay useful work SPEC Bechmarks Sldes based o a talk ad courtesy of Matthas Mueller, Ceter for Iformato Servces ad Hgh Performace Computg (ZIH) Techcal Uversty Dresde 3

14 What s SPEC? The Stadard Performace Evaluato Corporato (SPEC) s a o-proft corporato formed to establsh, mata ad edorse a stadardzed set of relevat bechmarks that ca be appled to the ewest geerato of hgh-performace computers. SPEC develops sutes of bechmarks ad also revews ad publshes submtted results from our member orgazatos ad other bechmark lcesees. For more detals see SPEC groups Ope Systems Group (desktop systems, hgh-ed workstatos ad servers) CPU (CPU bechmarks) JAVA (java clet ad server sde bechmarks) MAIL (mal server bechmarks) SFS (fle server bechmarks) WEB (web server bechmarks) Hgh Performace Group (HPC systems) OMP (OpeMP bechmark) HPC (HPC applcato bechmark) MPI (MPI applcato bechmark) Graphcs Performace Groups (Graphcs) Apc (Graphcs applcato bechmarks) Opc (OpeGL performace bechmarks) 4

15 Why do we eed bechmarks? Idetfy problems: measure mache propertes Tme evoluto: verfy that we make progress Coverage: help vedors to have represetatve codes Icrease competto by trasparecy Drve future developmet (see SPEC CPU2000) Relevace: help customers to choose the rght computer SPEC-Bechmarks All SPEC bechmarks are publcly avalable ad well kow/uderstood Compler ca troduce specal optmzatos for these bechmarks, whch mght be rrelevat for other, real-world applcatos. user has to provde the precse lst of comple-flags user has to provde performace of base (o-optmzed) ru Compler ca use statstcal formatos collected durg the frst executo order to optmze further rus (Cache ht rates, usage of regsters) Bechmarks desged such that exteral flueces are kept at a mumum (e.g. put/output) 5

16 SPEC CPU depedet programs: CINT teger bechmark: 2 applcatos ( C ad C++) CFP floatg-pot bechmark: 4 applcatos (6 Fortra-77, 4 Fortra-90 ad 4 C) Addtoal formato s avalable for each bechmark: Author of the bechmark Detalled descrpto Dokumetato regardg Iput ad Output Potetal problems ad refereces. CINT2000 6

17 CFP2000 Example for a CINT bechmark 7

18 Performace metrcs Two fudametally dfferet metrcs: speed rate (throughput) For each metrc results for two dfferet optmzato level have to be provded moderate optmzato aggressve optmzato 4 results for CINT results for CFP2000 = 8 metrcs If takg the measuremets of each applcato dvdually to accout: 2*2*(4+2)=04 metrcs 8

19 Performace metrcs (II) All results are relatve to a referece system The fal results s computed by usg the geometrc mea values Speed: Rate: SPEC SPEC ref t/ fp ( *00) t ru t t ref t/ fp ( *.6* N) t ru wth: : umber of bechmarks a sute t ref / t ru : executo tme for bechmark o the referece/test system N: Number of smultaeous tasks Reportg results SPEC produces a mmal set of represetatve umbers: + Reduces complexty to uderstad correlatos + Eases comparso of dfferet systems - Loss of formato Results have to be complat to the SPEC bechmakrg rules order to be approved as a offcal SPEC report All compoets have to avalable at least 3 moth after the publcato (cludg a rutme evromet for C/C++/Fortra applcatos) Usage of SPEC tools for complg ad reportg Each dvdual bechmark has to be executed at least three tmes Verfcato of the bechmark output A maxmum of four optmzato flags are allowed for the base ru (cludg preprocessor ad lk drectves) Dsclosure report cotag all relevat data has to be avalable 9

20 20

COSC 6385 Computer Architecture. Fundamentals

COSC 6385 Computer Architecture Fudametals Edgar Gabriel Sprig 208 Measurig performace (I) Respose time: how log does it take to execute a certai applicatio/a certai amout of work Give two platforms X