Lecture 5: Introduction to Entropy Coding. Thinh Nguyen Oregon State University

Transcription

1 Lecture 5: Introducton to Entropy Codng Thnh guyen Oregon State Unversty

2 Codes Defntons: Aphabet: s a coecton of symbos. Letters (symbos): s an eement of an aphabet. Codng: the assgnment of bnary sequences to eements of an aphabet. Code: A set of bnary sequences. Codeords: Indvdua members of the set of bnary sequences.

3 Exampes of Bnary Codes Engsh aphabets: 6 uppercase and 6 oercase etters and punctuaton mars. ASCII code for the etter a s 0000 ASCII code for the etter A s ASCII code for the etter, s 0000 ote: a the etters (symbos) n ths case use the same number of bts (7). These are caed fxed ength codes.

4 Exampes of Bnary Codes Engsh aphabets: 6 uppercase and 6 oercase etters and punctuaton mars. ASCII code for the etter a s 0000 ASCII code for the etter A s ASCII code for the etter, s 0000 ote: a the etters (symbos) n ths case use the same number of bts (7). These are caed fxed ength codes. The average number of bts per symbo (etter) s caed the rate of the code.

5 Code Rate Average ength of the code s mportant n compresson. Suppose our source aphabet conssts of four etters a, a, a 3, and a 4 th probabtes P(a ) 0.5 P(a ) 0.5, and P(a 3 ) P(a 4 ) 0.5. The average ength of the code s gven by 4 P( a ) n( a ) n(a ) s the number of bts n the codeord for etter a

6 Unquey Decodabe Codes Letters Probabtty Code Code Code 3 Code 4 a a a a Average Length Code : not unque a and a have the same codeord Code : not unquey decodabe: 00 coud mean a a 3 or a a a Codes 3 and 4: unquey decodabe: What are the rues? Code 3 s caed nstantaneous code snce the decoder nos the codeord the moment a code s compete.

7 Ho do e no a unquey decodabe code? Consder to codeords: 0 and 00 Prefx: 0 Dangng suffx: 0 Agorthm:. Construct a st of a the codeords.. Examne a pars of codeords to see f any codeord s a prefx of another codeord. If there exsts such a par, add the dangng suffce to the st uness there s one aready. 3. Contnue ths procedure usng the arger st unt:. Ether a dangng suffx s a codeord -> not unquey decodabe.. There are no more unque dangng suffxes -> unquey decodabe.

8 Exampes of Unque Decodabty Consder {0,0,} Dangng suffx s from 0 and 0 e st: {0,0,,} Dangng suffx s (from 0 and 0, and aso and ), and s aready ncuded n prevous teraton. Snce the dangng suffx s not a codeord, {0,0, } s unquey decodabe.

9 Exampes of Unque Decodabty Consder {0,0,0} Dangng suffx s from 0 and 0 e st: {0,0,0,} The ne dangng suffx s 0 (from 0 and ). Snce the dangng suffx 0 s a codeord, {0,0, 0} s not unquey decodabe.

10 Prefx Codes Prefx codes: A code n hch no codeord s a prefx to another codeord. A prefx code can be defned by a bnary tree Exampe:

11 Decodng a Prefx Codeord

12 Decodng a Prefx Codeord

13 Ho good s the code? Suppose a, b, and c occur th probabtes /8, /4, and 5/8, respectvey.

14 Are e osng any effcency by usng prefx code? The anser s O! Theorem : Let C be a code th code ords th engths,,. If C s unquey decodabe, then K( C) Theorem : Gven a set of ntegers,, that satsfy the nequaty e can aays fnd a prefx code th codeord engths,,.

15 Proof of Theorem ) ( C K n n n )... ( The exponent )... ( n s smpy the ength of n codeords Smaest vaue of s n and argest vaue s So, n n n A C K )] ( [ A s the number of combnatons of n codeords that have a combned ength of A Snce for a unquey decodabe code, each sequence can represent one and ony one sequence of codeords. Ths mpes )] ( [ n n A C K n n n n n Groth neary!!!! Thus, ) ( C K

16 Proof of Theorem : If e can aays fnd a prefx codes th the ength... Assume:..., Defne: 0, > Fact : bnary representaton of oud tae up )] ( [og ce Fact : The number of bts n the bnary representaton of s ess than og og ) ( og og

17 Proof of Theorem : If e can aays fnd a prefx codes th the ength..., o usng the bnary representaton of If ce (og ( )), e defne the codeord as:, then the th codeord c s the bnary representaton of If ce(og ( )), then the th codeord c s the bnary representaton of th ce(og ( )) zeros Ths s ceary a decodabe code ( are a dfferent snce s an ncreased functon, each ength ) aso has

18 Proof of Theorem : If e can aays fnd a prefx codes th the ength..., Suppose the cam s not true, then for some, < Ths means most sgnfcant bts fo form the bnary representon of c s the prefx of c, Hoever Therefore, That s the smaest vaue for s Hence, contradcts!