Burrows-Wheeler Transform and FM Index

Size: px

Start display at page:

Download "Burrows-Wheeler Transform and FM Index"

Maurice Wheeler
5 years ago
Views:

1 Burrows-Wheeler Trnsform nd M Index Ben ngmed You re free to use these slides. If you do, plese sign the guestbook ( or emil me (ben.lngmed@gmil.com) nd tell me briefly how you re using them. or originl Keynote files, emil me.

2 Burrows-Wheeler Trnsform Reversible permuttion of the chrcters of string, used originlly for compression b b $ T All rottions $ b b $ b b b $ b b $ b b b $ b $ b b b $ st column bb $ BWT(T) Sort Burrows-Wheeler Mtrix How is it useful for compression? How is it reversible? How is it n index? Burrows M, Wheeler DJ: A block sorting lossless dt compression lgorithm. Digitl Equipment Corportion, Plo Alto, CA 1994, Technicl Report 124; 1994

3 Burrows-Wheeler Trnsform def!rottions(t):!!!!"""!return!list!of!rottions!of!input!string!t!"""!!!!tt!=!t!*!2!!!!return![!tt[i:i+len(t)]!for!i!in!xrnge(0,!len(t))!]! def!bwm(t):!!!!"""!return!lexicogrphiclly!sorted!list!of!t s!rottions!"""!!!!return!sorted(rottions(t))! def!bwtvibwm(t):!!!!"""!given!t,!returns!bwt(t)!by!wy!of!the!bwm!"""!!!!return!''.join(mp(lmbd!x:!x[61],!bwm(t))) Mke list of ll rottions Sort them Tke lst column >>>!bwtvibwm("tomorrow_nd_tomorrow_nd_tomorrow$") 'w$wwdd nnoootttmmmrrrrrrooo ooo' >>>!bwtvibwm("it_ws_the_best_of_times_it_ws_the_worst_of_times$") 's$esttssfftteww_hhmmbootttt_ii woeeressii ' >>>!bwtvibwm('in_the_jingle_jngle_morning_ill_come_following_you$') 'u_gleeeengj_mlhl_nnnnt$nwj lggiolo_iiiirfcmylo_oo_' Python exmple:

4 Burrows-Wheeler Trnsform Chrcters of the BWT re sorted by their right-context This lends dditionl structure to BWT(T), tending to mke it more compressible Burrows M, Wheeler DJ: A block sorting lossless dt compression lgorithm. Digitl Equipment Corportion, Plo Alto, CA 1994, Technicl Report 124; 1994

5 Burrows-Wheeler Trnsform BWM bers resemblnce to the suffix rry $ b b $ b b b $ b b $ b b b $ b $ b b b $ BWM(T) 6 $ 5 $ 2 b $ 3 b $ 0 b b $ 4 b $ 1 b b $ SA(T) Sort order is the sme whether rows re rottions or suffixes

6 Burrows-Wheeler Trnsform In fct, this gives us new definition / wy to construct BWT(T): BWT[i] = T [SA[i] 1] if SA[i] > 0 $ if SA[i] =0 BWT = chrcters just to the left of the suffixes in the suffix rry $ b b $ b b b $ b b $ b b b $ b $ b b b $ BWM(T) 6 $ 5 $ 2 b $ 3 b $ 0 b b $ 4 b $ 1 b b $ SA(T)

7 Burrows-Wheeler Trnsform def!suffixarry(s):!!!!"""!given!t!return!suffix!rry!sa(t).!!we!use!python's!sorted!!!!!!!!function!here!for!simplicity,!but!we!cn!do!better.!"""!!!!stups!=!sorted([(s[i:],!i)!for!i!in!xrnge(0,!len(s))])!!!!#"extrct"nd"return"just"the"offsets!!!!return!mp(lmbd!x:!x[1],!stups) def!bwtvis(t):!!!!"""!given!t,!returns!bwt(t)!by!wy!of!the!suffix!rry.!"""!!!!bw!=![]!!!!for!si!in!suffixarry(t):!!!!!!!!if!si!==!0:!bw.ppend('$')!!!!!!!!else:!bw.ppend(t[si61])!!!!return!''.join(bw)!#"return"string4ized"version"of"list"bw Mke suffix rry Tke chrcters just to the left of the sorted suffixes >>>!bwtvis("tomorrow_nd_tomorrow_nd_tomorrow$") 'w$wwdd nnoootttmmmrrrrrrooo ooo' >>>!bwtvis("it_ws_the_best_of_times_it_ws_the_worst_of_times$") 's$esttssfftteww_hhmmbootttt_ii woeeressii ' >>>!bwtvis('in_the_jingle_jngle_morning_ill_come_following_you$') 'u_gleeeengj_mlhl_nnnnt$nwj lggiolo_iiiirfcmylo_oo_' Python exmple:

8 Burrows-Wheeler Trnsform How to reverse the BWT?? b b $ T All rottions $ b b $ b b b $ b b $ b b b $ b $ b b b $ st column bb $ BWT(T) Sort Burrows-Wheeler Mtrix BWM hs key property clled the Mpping...

9 Burrows-Wheeler Trnsform: T-rnking Give ech chrcter in T rnk, equl to # times the chrcter occurred previously in T. Cll this the T-rnking. 0 b b 1 3 $ Now let s re-write the BWM including rnks...

10 Burrows-Wheeler Trnsform BWM with T-rnking: $ 0 b b $ 0 b b b 1 3 $ 0 b 0 2 b 1 3 $ 0 b b b 1 3 $ b 1 3 $ 0 b b b 1 3 $ 0 ook t first nd lst columns, clled nd And look t just the s s occur in the sme order in nd. As we look down columns, in both cses we see: 3, 1, 2, 0

11 Burrows-Wheeler Trnsform BWM with T-rnking: $ 0 b b $ 0 b b b 1 3 $ 0 b 0 2 b 1 3 $ 0 b b b 1 3 $ b 1 3 $ 0 b b b 1 3 $ 0 Sme with bs: b 1, b 0

12 Burrows-Wheeler Trnsform Reversible permuttion of the chrcters of string, used originlly for compression b b $ T All rottions $ b b $ b b b $ b b $ b b b $ b $ b b b $ st column bb $ BWT(T) Sort Burrows-Wheeler Mtrix How is it useful for compression? How is it reversible? How is it n index? Burrows M, Wheeler DJ: A block sorting lossless dt compression lgorithm. Digitl Equipment Corportion, Plo Alto, CA 1994, Technicl Report 124; 1994

13 Burrows-Wheeler Trnsform: Mpping BWM with T-rnking: $ 0 b b $ 0 b b b 1 3 $ 0 b 0 2 b 1 3 $ 0 b b b 1 3 $ b 1 3 $ 0 b b b 1 3 $ 0 Mpping: The i th occurrence of chrcter c in nd the i th occurrence of c in correspond to the sme occurrence in T However we rnk occurrences of c, rnks pper in the sme order in nd

14 Burrows-Wheeler Trnsform: Mpping Why does the Mpping hold? Why re these s in this order reltive to ech other? $ b b 3 3 $ b b 1 1 b $ b 0 2 b $ b 1 0 b b $ b 1 $ b 2 b 0 b $ 0 $ b b 3 3 $ b b 1 1 b $ b 0 2 b $ b 1 0 b b $ b 1 $ b 2 b 0 b $ 0 Why re these s in this order reltive to ech other? They re sorted by right-context They re sorted by right-context Occurrences of c in re sorted by right-context. Sme for! Whtever rnking we give to chrcters in T, rnk orders in nd will mtch

15 Burrows-Wheeler Trnsform: Mpping BWM with T-rnking: $ 0 b b $ 0 b b b 1 3 $ 0 b 0 2 b 1 3 $ 0 b b b 1 3 $ b 1 3 $ 0 b b b 1 3 $ 0 We d like different rnking so tht for given chrcter, rnks re in scending order s we look down the / columns...

16 Burrows-Wheeler Trnsform: Mpping BWM with B-rnking: $ 3 b b $ 3 b b b 0 3 $ 3 b 1 2 b 0 0 $ 3 b b b 0 0 $ b 0 0 $ 3 b b b 0 0 $ 3 Ascending rnk now hs very simple structure: $, block of s with scending rnks, block of bs with scending rnks

17 Burrows-Wheeler Trnsform $ 0 0 b 0 1 b 1 Which BWM row begins with b 1? 2 1 Skip row strting with $ (1 row) 3 $ Skip rows strting with (4 rows) Skip row strting with b0 (1 row) row 6 b 0 b Answer: row 6

18 Burrows-Wheeler Trnsform Sy T hs 300 As, 400 Cs, 250 Gs nd 700 Ts nd $ < A < C < G < T Which BWM row (0-bsed) begins with G 100? (Rnks re B-rnks.) Skip row strting with $ (1 row) Skip rows strting with A (300 rows) Skip rows strting with C (400 rows) Skip first 100 rows strting with G (100 rows) Answer: row = row 801

19 Burrows-Wheeler Trnsform: reversing Reverse BWT(T) strting t right-hnd-side of T nd moving left Strt in first row. must hve $. contins chrcter just prior to $: 0 0 : Mpping sys this is sme occurrence of s first in. Jump to row beginning with 0. contins chrcter just prior to 0: b0. Repet for b 0, get 2 Repet for 2, get 1 Repet for 1, get b 1 Repet for b 1, get 3 $ b0 b1 0 b0 b1 1 $ 2 3 Repet for 3, get $, done Reverse of chrs we visited = 3 b b 0 0 $ = T

20 Burrows-Wheeler Trnsform: reversing Another wy to visulize reversing BWT(T): $ 0 $ 0 $ 0 $ 0 $ 0 $ 0 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ b 0 b 1 b 0 b 1 1 $ 2 3 T: 3 b b 0 0 $

21 Burrows-Wheeler Trnsform: reversing def;rnkbwt(bw): ;;;;''';Given;BWT;string;bw,;return;prllel;list;of;B6rnks.;;Also ;;;;;;;;returns;tots:;mp;from;chrcter;to;#;times;it;ppers.;''' ;;;;tots;=;dict() ;;;;rnks;=;[] ;;;;for;c;in;bw: ;;;;;;;;if;c;not;in;tots:;tots[c];=;0 ;;;;;;;;rnks.ppend(tots[c]) ;;;;;;;;tots[c];+=;1 ;;;;return;rnks,;tots def;firstcol(tots): ;;;;''';Return;mp;from;chrcter;to;the;rnge;of;rows;prefixed;by ;;;;;;;;the;chrcter.;''' ;;;;first;=;{} ;;;;totc;=;0 ;;;;for;c,;count;in;sorted(tots.iteritems()): ;;;;;;;;first[c];=;(totc,;totc;+;count) ;;;;;;;;totc;+=;count ;;;;return;first Clculte B-rnks nd count occurrences of ech chr Mke concise representtion of first BWM column def;reversebwt(bw): ;;;;''';Mke;T;from;BWT(T);''' ;;;;rnks,;tots;=;rnkbwt(bw) ;;;;first;=;firstcol(tots) ;;;;rowi;=;0;#;strt;in;first;row ;;;;t;=;'$';#;strt;with;rightmost;chrcter ;;;;while;bw[rowi];!=;'$': ;;;;;;;;c;=;bw[rowi] ;;;;;;;;t;=;c;+;t;#;prepend;to;nswer ;;;;;;;;#;jump;to;row;tht;strts;with;c;of;sme;rnk ;;;;;;;;rowi;=;first[c][0];+;rnks[rowi] ;;;;return;t Do reversl Python exmple:

22 Burrows-Wheeler Trnsform: reversing >>>!reversebwt("w$wwdd nnoootttmmmrrrrrrooo ooo") 'Tomorrow_nd_tomorrow_nd_tomorrow$' >>>!reversebwt("s$esttssfftteww_hhmmbootttt_ii woeeressii ") 'It_ws_the_best_of_times_it_ws_the_worst_of_times$' >>>!reversebwt("u_gleeeengj_mlhl_nnnnt$nwj lggiolo_iiiirfcmylo_oo_") 'in_the_jingle_jngle_morning_ill_come_following_you$' rnks list is m integers long! We ll fix lter. def;reversebwt(bw): ;;;;''';Mke;T;from;BWT(T);''' ;;;;rnks,;tots;=;rnkbwt(bw) ;;;;first;=;firstcol(tots) ;;;;rowi;=;0;#;strt;in;first;row ;;;;t;=;'$';#;strt;with;rightmost;chrcter ;;;;while;bw[rowi];!=;'$': ;;;;;;;;c;=;bw[rowi] ;;;;;;;;t;=;c;+;t;#;prepend;to;nswer ;;;;;;;;#;jump;to;row;tht;strts;with;c;of;sme;rnk ;;;;;;;;rowi;=;first[c][0];+;rnks[rowi] ;;;;return;t

23 Burrows-Wheeler Trnsform We ve seen how BWT is useful for compression: Sorts chrcters by right-context, mking more compressible string And how it s reversible: Repeted pplictions of Mpping, recreting T from right to left How is it used s n index?

24 M Index M Index: n index combining the BWT with few smll uxilliry dt structures M supposedly stnds for ull-text Minute-spce. (But inventors re nmed errgin nd Mnzini) Core of index consists of nd from BWM: cn be represented very simply (1 integer per lphbet chrcter) And is compressible Potentilly very spce-economicl! Polo errgin, nd Giovnni Mnzini. "Opportunistic dt structures with pplictions." oundtions of Computer Science, Proceedings. 41st Annul Symposium on. IEEE, $ b b $ b b b $ b b $ b b b $ b $ b b b $ Not stored in index

25 M Index: querying Though BWM is relted to suffix rry, we cn t query it the sme wy $ b b $ b b b $ b b $ b b b $ b $ b b b $ 6 $ 5 $ 2 b $ 3 b $ 0 b b $ 4 b $ 1 b b $ We don t hve these columns; binry serch isn t possible

26 M Index: querying ook for rnge of rows of BWM(T) with P s prefix Do this for P s shortest suffix, then extend to successively longer suffixes until rnge becomes empty or we ve exhusted P Esy to find ll the rows beginning with, thnks to s simple structure P = b $ b b 3 0 $ b b 1 1 b $ b 0 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 0

27 M Index: querying We hve rows beginning with, now we seek rows beginning with b P = b $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 ook t those rows in. b 0, b 1 re bs occuring just to left. Use Mpping. et new rnge delimit those bs P = b $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 Now we hve the rows with prefix b

28 M Index: querying We hve rows beginning with b, now we seek rows beginning with b P = b $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 Use Mpping 2, 3 occur just to left. P = b $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 Now we hve the rows with prefix b

29 M Index: querying P = b Now we hve the sme rnge, [3, 5), we would hve got from querying suffix rry [3, 5) Where re these? $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 [3, 5) 6 $ 5 $ 2 b $ 3 b $ 0 b b $ 4 b $ 1 b b $ Unlike suffix rry, we don t immeditely know where the mtches re in T...

30 M Index: querying When P does not occur in T, we will eventully fil to find the next chrcter in : Rows with b prefix P = bb b $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 No bs!

31 M Index: querying If we scn chrcters in the lst column, tht cn be very slow, O(m) P = b $ b b 3 0 $ b b 1 1 b $ b 0 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 0 Scn, looking for bs

32 M Index: lingering issues (2) Storing rnks tkes too much spce (1) Scnning for preceding chrcter is slow $ b b 0 0 $ b b0 1 b $ b1 2 b $ b 1 3 b b $ b0 $ b 2 O(m) scn (3) m integers def;reversebwt(bw): ;;;;""";Mke;T;from;BWT(T);""" ;;;;rnks,;tots;=;rnkbwt(bw) ;;;;first;=;firstcol(tots) ;;;;rowi;=;0 ;;;;t;=;"$" ;;;;while;bw[rowi];!=;'$': ;;;;;;;;c;=;bw[rowi] ;;;;;;;;t;=;c;+;t ;;;;;;;;rowi;=;first[c][0];+;rnks[rowi] ;;;;return;t Need wy to find where mtches occur in T: b1 b $ 3 Where? $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3

33 M Index: fst rnk clcultions Is there n O(1) wy to determine which bs precede the s in our rnge? $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 Ide: pre-clculte # s, bs in up to every row: $ b b b b $ Tlly b We infer b0 nd b1 pper in in this rnge O(1) time, but requires m integers

34 M Index: fst rnk clcultions Another ide: pre-clculte # s, bs in up to some rows, e.g. every 5 th row. Cll pre-clculted rows checkpoints. $ b b b b $ Tlly b ookup here succeeds s usul Oops: not checkpoint But there s one nerby To resolve lookup for chrcter c in non-checkpoint row, scn long until we get to nerest checkpoint. Use tlly t the checkpoint, djusted for # of cs we sw long the wy.

35 M Index: fst rnk clcultions Checkpoint Wht s my rnk? = 483 s long the wy Assuming checkpoints re spced O(1) distnce prt, lookups re O(1) tlly -> rnk Wht s my rnk? = b b b b b b b b Tlly b

36 M Index: few problems Solved! At the expense of dding checkpoints (O(m) integers) to index. (1) (2) Rnking tkes too much spce $ b b 0 0 $ b b0 1 b $ b1 2 b $ b 1 3 b b $ b0 $ b 2 b1 b $ 3 With checkpoints it s O(1) This scn is O(m) work m integers def;reversebwt(bw): ;;;;""";Mke;T;from;BWT(T);""" ;;;;rnks,;tots;=;rnkbwt(bw) ;;;;first;=;firstcol(tots) ;;;;rowi;=;0 ;;;;t;=;"$" ;;;;while;bw[rowi];!=;'$': ;;;;;;;;c;=;bw[rowi] ;;;;;;;;t;=;c;+;t ;;;;;;;;rowi;=;first[c][0];+;rnks[rowi] ;;;;return;t With checkpoints, we gretly reduce # integers needed for rnks But it s still O(m) spce - there s literture on how to improve this spce bound

37 M Index: few problems Not yet solved: (3) Need wy to find where $ b b 0 these occurrences re in T: 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ If suffix rry were prt of index, we b 0 $ b 2 b 1 b $ 3 could simply look up the offsets $ b b $ b b b $ b b $ b b b $ b $ b b b $ Offsets: 0, 3 SA 6 $ 5 $ 2 b $ 3 b $ 0 b b $ 4 b $ 1 b b $ But SA requires m integers

38 M Index: resolving offsets Ide: store some, but not ll, entries of the suffix rry $ b b $ b b b $ b b $ b b b $ b $ b b b $ X SA ookup for row 4 succeeds - we kept tht entry of SA ookup for row 3 fils - we discrded tht entry of SA

39 M Index: resolving offsets But Mpping tells us tht the t the end of row 3 corresponds to......the t the begining of row 2 $ b b $ b b b $ b b $ b b b $ b $ b b b $ SA And row 2 hs suffix rry vlue = 2 So row 3 hs suffix rry vlue =???? 3 = 2 (row 2 s SA vl) + 1 (# steps to row 2) If sved SA vlues re O(1) positions prt in T, resolving offset is O(1) time

40 M Index: problems solved Solved! At the expense of dding some SA vlues (O(m) integers) to index Cll this the SA smple (3) Need wy to find where these occurrences re in T: $ b b 0 0 $ b b 0 1 b $ b 1 2 b $ b 1 3 b b $ b 0 $ b 2 b 1 b $ 3 With SA smple we cn do this in O(1) time per occurrence

41 M Index: smll memory footprint Components of the M Index: irst column (): st column (): SA smple: Checkpoints: ~ integers m chrcters m integers, where is frction of rows kept m b integers, where b is frction of rows checkpointed Exmple: DNA lphbet (2 bits per nucleotide), T = humn genome, = 1/32, b = 1/128 irst column (): st column (): SA smple: Checkpoints: 16 bytes 2 bits * 3 billion chrs = 750 MB 3 billion chrs * 4 bytes/chr / 32 = ~ 400 MB 3 billion * 4 bytes/chr / 128 = ~ 100 MB Totl < 1.5 GB

42 Approximte Serch

43 Bowtie 2

44 BWT-bsed ssembly Problem: How would you use the BWT to find n overlp lignment of length l?

Cache CPI and DFAs and NFAs. CS230 Tutorial 10

Cache CPI and DFAs and NFAs. CS230 Tutorial 10 Cche CPI nd DFAs nd NFAs CS230 Tutoril 10 Multi-Level Cche: Clculting CPI When memory ccess is ttempted, wht re the possible results? ccess miss miss CPU L1 Cche L2 Cche Memory L1 cche hit L2 cche hit