Controlling a population of identical MDP

Transcription

1 Controlling popultion of identicl MDP Nthlie Bertrnd Inri Rennes ongoing work with Miheer Dewskr (CMI), Blise Genest (IRISA) nd Hugo Gimert (LBRI) Trends nd Chllenges in Quntittive Verifiction Mysore, Ferury 2nd 2016

2 Motivtion Control of gene expression for popultion of cells credits: G. Btt Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

3 Motivtion Control of gene expression for popultion of cells cell popultion gene expression monitored through fluorescence level drug injections ffect ll cells response vries from cell to cell credits: G. Btt otin lrge proportion of cells with desired gene expression level Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

4 Motivtion Control of gene expression for popultion of cells cell popultion gene expression monitored through fluorescence level drug injections ffect ll cells response vries from cell to cell credits: G. Btt otin lrge proportion of cells with desired gene expression level ritrry n of components full oservtion uniform control MDP model for single cell glol quntittive rechility ojective Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

5 Mrkov decision processes,1/2, non-deterministic ctions: {, } pro. distriution over successors,1/2 F Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

6 Mrkov decision processes,1/2, non-deterministic ctions: {, } pro. distriution over successors,1/2 F Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

7 Mrkov decision processes,1/2, non-deterministic ctions: {, } pro. distriution over successors,1/2 F Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

8 Mrkov decision processes,1/2, non-deterministic ctions: {, } pro. distriution over successors,1/2 F Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

9 Mrkov decision processes,1/2,1/2, non-deterministic ctions: {, } pro. distriution over successors Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

10 Mrkov decision processes,1/2,1/2, non-deterministic ctions: {, } pro. distriution over successors Scheduler σ : S + Σ resolves non-determinism induces Mrkov chin with proility mesure P σ Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

11 Mrkov decision processes,1/2,1/2, non-deterministic ctions: {, } pro. distriution over successors Scheduler σ : S + Σ resolves non-determinism induces Mrkov chin with proility mesure P σ Theorem: rechility checking for MDP The following prolems re in PTIME σ, P σ ( F ) = 1? σ, P σ ( F ) >.7? compute mx σ P σ ( F ). Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

12 Bck to our motivting ppliction Control of gene expression for popultion of cells credits: G. Btt ritrry n of components full oservtion uniform control MDP model for single cell glol quntittive rechility ojective Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

13 Modelling popultion of N identicl MDP M uniform control policy under full oservtion Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

14 Modelling popultion of N identicl MDP M uniform control policy under full oservtion,1/2, F,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

15 Modelling popultion of N identicl MDP M uniform control policy under full oservtion,1/2,,1/2, F,1/2,1/4,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

16 Modelling popultion of N identicl MDP M uniform control policy under full oservtion,1/2,,1/2, F,1/2,1/4,1/2 Verifiction question does the mximum proility tht given proportion of MDPs rech trget set of sttes meet threshold? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

17 Modelling popultion of N identicl MDP M uniform control policy under full oservtion,1/2,,1/2, F,1/2,1/4,1/2 Verifiction question does the mximum proility tht given proportion of MDPs rech trget set of sttes meet threshold? Fixed N: uild the product MDP M N, identify glol trget sttes, compute optiml scheduler Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

18 Prmeterized verifiction Verifiction question does the mximum proility tht given proportion of MDPs rech trget set of sttes meet threshold? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

19 Prmeterized verifiction Verifiction question does the mximum proility tht given proportion of MDPs rech trget set of sttes meet threshold? Prmeter N: check the glol ojective for ll popultion sizes N N mx σ P σ(m N = t lest 80% of MDPs in F ).7? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

20 Prmeterized verifiction Verifiction question does the mximum proility tht given proportion of MDPs rech trget set of sttes meet threshold? Prmeter N: check the glol ojective for ll popultion sizes N N mx σ P σ(m N = t lest 80% of MDPs in F ).7? Restricted cses qulittive: lmost-sure convergence Boolen: sure convergence N mx σ P σ(m N = F N )= 1? N σ, M N = σ F N? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

21 This tlk Prolem setting Boolen prmeterized verifiction questions uniform control for popultion of NFA 2-plyer turn-sed gme controller chooses the ction (e.g. ) opponent chooses how to move ech individul copy (-trnsition) convergence ojective: ll copies in trget set F Q N σ, M N = σ F N? N σ, τ, (M N, σ, τ) = F N? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

22 This tlk Prolem setting Boolen prmeterized verifiction questions uniform control for popultion of NFA 2-plyer turn-sed gme controller chooses the ction (e.g. ) opponent chooses how to move ech individul copy (-trnsition) convergence ojective: ll copies in trget set F Q N σ, M N = σ F N? N σ, τ, (M N, σ, τ) = F N? Questions ddressed decidility memory requirements for controller σ dmissile vlues for N Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

23 Monotonicity property, N σ, M N = σ F N? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

24 Monotonicity property, N σ, M N = σ F N? Monotonicity: hrder when N grows σ, M N = σ F N = M N, σ, M M = σ F M Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

25 Monotonicity property, N σ, M N = σ F N? Monotonicity: hrder when N grows σ, M N = σ F N = M N, σ, M M = σ F M Cutoff: smllest N for which there is no dmissile controller σ Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

26 A first exmple nd first question, F N, σ, M N = σ F N σ(k, 0, 0, ) = σ(0, k u, k d, ) = σ(0, 0, k d, ) = memoryless support-sed controllers suffice on this exmple Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

27 A first exmple nd first question, F N, σ, M N = σ F N σ(k, 0, 0, ) = σ(0, k u, k d, ) = σ(0, 0, k d, ) = memoryless support-sed controllers suffice on this exmple Question 1 Are memoryless support-sed controllers enough in generl? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

28 A second exmple nd second question A = { 1,, M } unspecified edges led to sink stte q 1 A\ 1. F A {} q M A\ M N < M, σ, M N = σ F N Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

29 A second exmple nd second question A = { 1,, M } unspecified edges led to sink stte q 1 A\ 1. F A {} q M A\ M N < M, σ, M N = σ F N Cutoff min{n σ, M N = σ F N } here O( M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

30 A second exmple nd second question A = { 1,, M } unspecified edges led to sink stte q 1 A\ 1. F A {} q M A\ M N < M, σ, M N = σ F N Cutoff min{n σ, M N = σ F N } here O( M ) Question 2 Are cutoffs lwys polynomil in M? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

31 A first nswer Assumption: if t lest one stte is empty, the controller ensures convergence gdget similr to previous exmple with ctions { 1,, 4} Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

32 A first nswer Assumption: if t lest one stte is empty, the controller ensures convergence gdget similr to previous exmple with ctions { 1,, 4} Possile controllers lwys : deterministic ehviour, full support is mintined lwys : splitting the copies in third stte llows opponent to win Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

33 A first nswer Assumption: if t lest one stte is empty, the controller ensures convergence gdget similr to previous exmple with ctions { 1,, 4} Possile controllers lwys : deterministic ehviour, full support is mintined lwys : splitting the copies in third stte llows opponent to win nd in lterntion: lek from first/second sttes to third Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

34 A first nswer Assumption: if t lest one stte is empty, the controller ensures convergence gdget similr to previous exmple with ctions { 1,, 4} Possile controllers lwys : deterministic ehviour, full support is mintined lwys : splitting the copies in third stte llows opponent to win nd in lterntion: lek from first/second sttes to third Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

35 A first nswer Assumption: if t lest one stte is empty, the controller ensures convergence gdget similr to previous exmple with ctions { 1,, 4} Possile controllers lwys : deterministic ehviour, full support is mintined lwys : splitting the copies in third stte llows opponent to win nd in lterntion: lek from first/second sttes to third Memoryless support-sed controllers re not enough! Exponentil memory on top of support my even e needed. Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

36 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

37 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

38 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

39 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

40 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

41 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

42 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

43 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

44 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

45 A second nswer d u c c u d F,,c u,d u,d u,d,,c 2M ottom sttes (here 6) N 2 M, σ, M N = σ F N ccumulte copies in ottom sttes, then u/d to converge for N > 2 M controller cnnot void reching the sink stte Cutoff cn even e douly exponentil! Cutoff O(2 M ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

46 Lessons lernt so fr Boolen prolem is hrder thn expected Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

47 Lessons lernt so fr Boolen prolem is hrder thn expected supports re not enough douly exponentil lower ound on cutoffs somehow prevents from uilding the product MDP Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

48 Lessons lernt so fr Boolen prolem is hrder thn expected supports re not enough douly exponentil lower ound on cutoffs somehow prevents from uilding the product MDP the more copies the hrder, the lrger support the hrder looking t whether supports cn e mintined seems promising Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

49 Support gme Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

50 Support gme plyer gme on possile supports Eve chooses ction Adm chooses trnsfer reltion 1,2,3 1,2,3,4 1,3,4 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

51 Support gme plyer gme on possile supports Eve chooses ction Adm chooses trnsfer reltion 1,2,3 1,2,3,4 simple winning condition for Eve: rech {F } sufficient condition, not sound in generl 1,3,4 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

52 Refined winning condition Intuition: llow Eve to monitor some copies nd pinpoint leks long ply only finitely mny leks re possile Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

53 Refined winning condition Intuition: llow Eve to monitor some copies nd pinpoint leks long ply only finitely mny leks re possile Ply ρ = S 1 R 0 1 S1 winning for Eve if there exists (T i ) i N s.t. (1) i, T i S i (2) i, Pre[R i+1 ](T i+1 ) T i (3) j, T j+1 Post[R j+1 ](T j ) w S i 1,2,3,4 w S i+1 w S i+2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

54 Refined winning condition Intuition: llow Eve to monitor some copies nd pinpoint leks long ply only finitely mny leks re possile Ply ρ = S 1 R 0 1 S1 winning for Eve if there exists (T i ) i N s.t. (1) i, T i S i (2) i, Pre[R i+1 ](T i+1 ) T i (3) j, T j+1 Post[R j+1 ](T j ) w S i 1,2,3,4 w S i+1 w S i+2 Eve wins support gme with refined winning condition iff N controller hs strtegy to rech winning supports Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

55 Solving support gme w. refined winning condition Trnsformtion into 2-plyer prtil oservtion gme with Büchi winning condition exponentil lowup of gme ren sttes (S, T ) for ll possile T S Adm shll not oserve the susets monitored y Eve he only oserves S-component of stte (S, T ) Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

56 Solving support gme w. refined winning condition Trnsformtion into 2-plyer prtil oservtion gme with Büchi winning condition exponentil lowup of gme ren sttes (S, T ) for ll possile T S Adm shll not oserve the susets monitored y Eve he only oserves S-component of stte (S, T ) Theorem: Decidility nd complexity (still to e checked) Boolen prmeterized convergence is decidle in 3EXPTIME. Cutoff is t most triply exponentil in M. Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

57 Solving support gme w. refined winning condition Trnsformtion into 2-plyer prtil oservtion gme with Büchi winning condition exponentil lowup of gme ren sttes (S, T ) for ll possile T S Adm shll not oserve the susets monitored y Eve he only oserves S-component of stte (S, T ) Theorem: Decidility nd complexity (still to e checked) Boolen prmeterized convergence is decidle in 3EXPTIME. Cutoff is t most triply exponentil in M. Theorem: (fr from mtching) Lower-ounds PSPACE-hrdness for Boolen prmeterized convergence. Douly exponentil lower-ound on the cutoff. Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

58 Contriutions Uniform control of popultion of identicl MDP prmeterized verifiction prolem Boolen convergence: ring ll MDP t the sme time in F surprisingly quite involved! eyond support-sed optiml controllers 3EXPTIME-decision procedure cutoff etween douly exponentil nd triply exponentil Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

59 Bck to motivtions Motivtion 1: prcticl motivtion Control of gene expression for popultion of cells credits: G. Btt Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

60 Bck to motivtions Motivtion 1: prcticl motivtion Control of gene expression for popultion of cells credits: G. Btt need for truely proilistic model MDP insted of NFA need for truely quntittive questions proportions nd proilities insted of convergence nd (lmost)-sure N mx σ P σ(m N = t lest 80% of MDPs in F ).7? Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

61 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

62 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

63 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

64 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2,1/2,1/2 Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

65 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2,1/2,1/2 Argule: optiml rechility proility not continuous when N,,1/2 d u F,1/2 d u Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

66 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2,1/2,1/2 Argule: optiml rechility proility not continuous when N,,1/2,1/2 u d d F N, σ, P σ ( F N ) = 1. In the PA, the mximum proility to rech F is.5. u Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

67 Bck to motivtions Motivtion 2: theoreticl motivtion Discrete pproximtion of proilistic utomt,1/2,1/2,1/2,1/2 Argule: optiml rechility proility not continuous when N,,1/2,1/2 u d d F N, σ, P σ ( F N ) = 1. In the PA, the mximum proility to rech F is.5. u Good news? hope for lterntive more decidle semntics for PA Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20

68 Thnks for your ttention! Controlling popultion of MDP Nthlie Bertrnd Mysore, Ferury 2nd / 20