A Regime Switching Independent Component Analysis Method for Temporal Data

Transcription

1 Journal of Compuaions & Modelling, vol.2, no.1, 2012, ISSN: (prin), (online) Inernaional Scienific Press, 2012 A Regime Swiching Independen Componen Analysis Mehod for Temporal Daa Ho-Yin Yue 1,2 Absrac A mixure of independen componen analysis mehod for emporal daa is presened in his paper. The mehod is derived by modeling he observaions as a mixure of ICA (mica). mica model has been applied o daa classificaion and image processing. However, i is hard o use mica in assigning class memberships of emporal daa. In he proposed mehod, memberships of he daa are modified according o is pas values in he learning process. I shows ha he proposed mehod is able o deec he swich beween mixures in highly overlapped daa, which have smaller error han radiional mica mehod. Mahemaics Subjec Classificaion: 68Q32, 68T10 Keywords: auomaic conex swiching, blind source separaion, independen componen analysis, mixure model 1 Deparmen of Compuing, Hang Seng Managemen College, Hong Kong, willyyue@gmail.com 2 School of Finance, Shanghai Universiy of Finance and Economics, Shanghai, China Aricle Info: Received : February 28, Revised : March 27, 2012 Published online : April 10, 2012

2 110 Independen Componen Analysis for Temporal Daa 1 Inroducion In he las decade, Independen Componen Analysis (ICA) [1] [2] became a ho opic in he field of signal processing and daa mining. The aim of ICA is decomposing he observaions linearly ino a se of independen componens which are saisically independen. The marix maps he observaions o he independen componens is called demixing marix and is inverse is mixing marix. In radiional ICA mehod, boh he mixing and demixing marices are assumed o be consan. This assumpion implies ha he environmen is unchanged hroughou he ime. However, real environmen eeps changing, so he mixing and demixing marices change wih ime. To model he dynamic of he mixing marix, non-saionary independen componen analysis [3] [4] and hidden Marov independen componen analysis (HMICA) [5] [6] are wo commonly used approaches. These wo models have differen assumpions abou how observaions are mixed from he independen sources. Assume here are m m independen sources whose probabiliy densiy funcions are p ( s ). The non-saionary ICA assumes ha he sources are mixed linearly by he mixing marix A wih observaional noise assumed o change wih he pas observaions ( w (Equaion 1). The mixing marix m A is X ) according o Equaion 2. The graphical model describes non-saionary ICA is shown in Figure 1. X AS w (1) vec( A ) F v 1 P( ) px ( ) p( ) px ( ) 1 (2) where, v is zero-mean Gaussian noise wih covariance Q, F is he sae ransiion marix, and denoes he collecion of observaions {,,, }. X1 X2 X

3 Ho-Yin Yue 111 Figure 1: Generaive model of he non-saionary ICA model In he HMICA, every observaion a ime insan belong o a sae q. Each sae is associaed wih he mixing marix source parameers vecor S 1 A, he demixing marix W and he. Wih he source generaed a ime is given as f( q, S ). The observaion a ime is generaed as X AS. Figure 2 shows he general model of he HMICA graphically. Figure 2: Generaive model of he HMICA model

4 112 Independen Componen Analysis for Temporal Daa To summarize non-saionary ICA and HMICA, boh models have he same general form, X AS, which is assumed ha he mixing marix ( A ) is changing wih ime. The main difference beween hem is he assumpion on he dynamic of he mixing marix. The non-saionary ICA assumes ha he mixing marix is a funcion of pas observaions. The HMICA assumes ha he mixing marix changes in a form of Hidden Marov Model. However, neiher he non-saionary ICA nor he HMICA can model he observaions well when observaions are generaed by a mixure of co-exising sysems. In his paper, a Temporal Mixure of Independen Componen Analysis (mica) mehod is suggesed o model his ind of mixure sysem. In he proposed mehod, memberships of he daa are modified according o is pas values in he learning process. I shows ha he proposed mehod is able o deec he swich beween mixures in highly overlapped daa, which have smaller error han radiional mica mehod. This paper is organized as follow: In Secion 2, radiional mixure of ICA modeling was presened. An independen componen analysis mehod for emporal daa is proposed in secion 3. Secion 4 and 5 are he experimens and conclusion of he paper respecively. 2 Mixure of ICA Modeling Tradiional ICA [7] allows only one mixing marix in he sysem. I is unable for radiional ICA mehod o decompose he observaions correcly if observaions are produced by several co-exising mixing/demixing sysem. mica [8] relaxed he radiional ICA by assuming ha he observaions are generaed from sources in more han one mixing sysems; eeping he sources wihin he same mixing sysem saisically independen wih ohers a he same ime. Using mica, some applicaions have been buil in image processing [9] [10] and daa clusering [11].

5 Ho-Yin Yue 113 In mica, he daa Xn x1, n, x2, n,, xm, n is m dimensional observaion a ime n generaed by a mixure densiy model [12]. The probabiliy of generaing a daa poin, from a K-componen mixure model is: K n 1 n c p ( X ) p ( X C, ) p ( C ) (3) Also, he probabiliy ha X n is generaed from componen px ( n C, ) pc ( ) pc ( i Xn) K p( X, C ) p( C ) where, 1 n T C i is: (4) is he vecor of unnown parameers for h mixure. C denoes he h mixure and he number of mixure K is assumed o be nown in advance. Daa wihin h mixure is described by he sandard ICA model: X n AS, n (5) where A is a m m mixing marix and Sn, s,1, n, s,2, n,, smn,, is he m dimensional source for h mixure respecively. I is shown ha he model parameers can be esimaed by maximizing he sum of he log-lielihood of he daa (Equaion 6) hrough Expecaion-Maximizaion (EM) algorihm [11]. N K (6) L log p( X C, ) ( ) 1 1 n p C n A survey of mica was given in [13]. The main difficuly for applying mica on emporal daa is ha a single daa does no conain enough informaion for assigning he class membership. This problem is he mos serious when observaions having similar membership values among differen mixure. In his paper, a emporal mica mehod, mica hereafer, is proposed. In mica, memberships for differen ICA mixures are used o model he changes of sysem gaing. Tae cocail pary as an example, when he microphone was moved from one posiion o anoher posiion a ime, he mixing marix would be changed a ime wih he posiion of he microphone for he same sources. In order o model T

6 114 Independen Componen Analysis for Temporal Daa such a case wih mica, wo ICA mixures were used. Suppose A 1 and A 2 are mixing marices used in mica. Given any observaion, is membership of A1 is greaer han membership of A 2 before ime, and he membership of A 1 becomes lower han A 2 afer ime. In oher words, he change of he mixing marix can be indicaed by he change of membership value. 3 Temporal Mixure of ICA Modeling In his secion, descripion on he learning process of mica is presened. Given a m dimensional observaions, Xn x1, n, x2, n,, xm, n generaed by a K-componen mixures densiy model. Probabiliy of generaed from mixure described by he sandard ICA model: T, which are X being C is given by Equaion 4. For he h mixure, daa is X AS (7), In mica, p( C X ) is used o represen he membership of he observaion X o he h ICA mixure a ime. The deails of he source model used o calculae p( C X ) is given in [7]. Memberships are assumed o change smoohly across ime. Therefore, afer obained p( C X ), we smooh he value by: pc ( X L ) smooh ( i ) pc ( X ) i1 i (8) where, () i 0 is a consan ha represens he imporance for affecing, pc ( X ) and p( C X ) wih he consrain i L () i 1. However, smoohing he membership probabiliy is no enough for obaining a sable resul. In mica, Smoohed probabiliies are furher modified o decrease he effec of ambiguiy p( C X ) in he membership assignmen. i1

7 Ho-Yin Yue 115 Afer he memberships are smoohed by Equaion 8, memberships are modified as follow: if pc ( X) smooh pc ( i X) smooh,i hen, pc ( X) modified pc ( X) smooh. pc ( X) pc ( X),i hen, if smooh i smooh 1 pc ( X) modified pc ( X) smooh where, 1 is modificaion facor. Then, normalizaion is performed according o Equaion 9 in order o eep he consrain pc ( i X) 1. K i1 pc ( X) norm K pc ( X) modified (9) pc ( X) i1 i modified Afer assigned he membership values, hese modified memberships are used ogeher wih FasICA [15][16] o calculae he mixing marix in each mixure. The algorihm of mica is oulined in Algorihm 1. In he nex secion, mica are esed wih sources which are highly overlapped. Algorihm 1: Learning algorihm for mica. Inpu: A M-dimensional emporal daa Oupu: A se N-dimensional of esimaed sources, N X, and he number of ICA mixures K. N mixing marices of each ICA mixure, and membership probabiliies for every observaion in each ICA mixure. Sep 1: Choose an iniial (e.g. random) demixing marix B( ) for all mixures. Sep 2: Compue he independen componens ( S ) and he membership probabiliy ( p( C X )) for each pair of he h mixure and observaion a ime.

8 116 Independen Componen Analysis for Temporal Daa 2.1. for all mixures S, B X 2.2. for all pairs of mixure and ime. Sep 3: Modify he membership probabiliy for each pair of he h mixure and observaion a ime L pc ( X) ( i) pc ( X ) i1 i 3.2. if p( C X ) p( C X ), i hen, i p( C X ) p( C X ). if p( C X ) p( C X ), i hen, i 1 p( C X) p( C X) pc ( X) 3.3 pc ( X) K ( ) i p C 1 i X Sep 4: Perform FasICA algorihm for each mixure. Cener he daa o mae is mean zero. Compue he weighed correlaion marix (C ) for each mixure. 4.1 i, Esi,, si,, { anh( )} i 1, 2,, M. 4.2 i 1, 2,, M. 2 i, 1/( i, E{1 anh( si,, ) }) Updae he separaing marix by: T 4.3. B Bdiag i, diag i, E si,, si,, B ( )[ ( ) {anh( ) } ] Decorrelae and normalize he separaing marix by: 4.4 B ( BCB ) B T 1/2 Sep 5: If no converged, go bac o Sep 2.

9 Ho-Yin Yue Experimenal Resuls In his secion, mica was applied on a se of emporal daa from ICA mixure wih wo componens. The same 2D sources are used in boh mixures. One dimension of he sources is riangular wave while anoher is sine wave. The sources were shown in Figure 3. Figure 3: The sources used o generae he observaions 1000 observaions were generaed in he sense ha he firs 500 observaions were generaed from he firs mixure while he remaining 500 observaions were generaed from anoher mixure. The observaions from he mixure were shown in Figure 4. Figure 4: The observaions generaed from he ICA mixure

10 118 Independen Componen Analysis for Temporal Daa From he scaer plo of he observaions (Figure 5), i shows ha he observaions from he mixure are highly overlapped. So, o idenify he ruh memberships and he sources from he observaions are difficul. Figure 5: The scaer plo of he observaions The sources of he mixures which recovered from mica were shown in Figure 6 and Figure 7. Membership resuls show ha he firs ICA mixure dominaes o he firs 500 observaions and he second ICA mixure dominaes o he remaining 500 observaions. The sources recovered from mica are very close o he sources used o generae he observaions. The resuls of mica are compared wih mica mehod [13]. Resuls of mica are shown in Figure 8 and Figure 9. I is found ha alhough boh mehods are capable o decide he membership probabiliy correcly. mica produces a more accurae recovered sources han mica mehod. So, i is

11 Ho-Yin Yue 119 concluded ha mica has comparaively beer sources recovery power in daa which observaions generaed from highly overlapped mixures. Figure 6: The independen componens and he membership for he firs mixure wih mica Figure 7: The independen componens and he membership for he second mixure wih mica 5 Conclusion An independen componen analysis mehod for emporal daa, mica, is presened in his paper. In mica, observaions are modeled as a mixure of sysems. Each sysem is furher described by an ICA model. Memberships of observaions are used o decide he degree of influence for a mixure owards he

12 120 Independen Componen Analysis for Temporal Daa observaions. When compared wih mica model, memberships in mica are modified in he esimaion process. This modificaion provides a beer assignmen in he memberships. According o he experimenal resuls, mica shows a excellen power in discovering he conex swich of observaions auomaically in highly overlapping daa. The esimaed sources are closer o he rue sources when compare wih hose esimaed sources from ICA mixure model [8]. Figure 8: The independen componens and he membership for he firs mixure wihou mica Figure 9: The independen componens and he membership for he second mixure wihou mica

13 Ho-Yin Yue 121 References [1] S. Amari and J. Cardoso, Blind source separaion semiparameric saisical approach, IEEE Transacions on Signal Processing, 5(11), (1997), [2] A. Bell and T. Sejnowsi, An informaion-maximizaion approach o blind separaion and blind deconvoluion, Neural Compuaion, 7, (1995), [3] R. Everson and S. Robers, Non-saionary independen componen analysis, ICANN-99, (1999). [4] R. Everson and S. Robers, Paricle filers for non-saionary ICA, Advances in independen componens analysis, M. Girolami, Ed. Springer, [5] W. Penny, R. Everson and S. Robers, Hidden Marov independen componens analysis, Advances in independen componens analysis, M. Girolami, Ed. Springer, [6] W. Penny, S. Robers and R. Everson, Hidden Marov independen componens for biosignal analysis, MEDSIP-2000, Inernaional Conference on Advances in Medical Signal and Informaion Processing 2000, Brisol, UK, (Sepember 2000). [7] A. Hyvärinen, J. Karhunen and E. Oja, Independen Componen Analysis, John Wiley & Sons, , [8] T. Lee, M. Lewici and T. Sejnowsi, Unsupervised classificaion wih non-gaussian mixure models using ica, Proceedings of he 1998 conference on Advances in neural informaion processing sysems II, (1999), [9] C. Shah, M. Arora, S. Robila and P. Varshney, ICA mixure model based unsupervised classificaion of hyperspecral imagery, Proceedings of he 31s Applied Imagery Paern Recogniion Worshop (AIPR 02), (2002), [10] T. Lee and M. Lewici, Unsupervised image classificaion, segmenaion, and enhancemen using ica mixures models, IEEE Transacaions on Image Processing, 11(3), (2002),

14 122 Independen Componen Analysis for Temporal Daa [11] S. Robers and W. Penny, Mixures of independen componen analysers, Proceedings of he Inernaional Conference on Arificial Neural Newors, (2001), [12] R. Duda and P. Har, Paern classificaion and scene analysis, Wiley, [13] T. Lee, M. Lewici and T. Sejnowsi, ICA mixure models for unsupervised classificaion of non-gaussian classes and auomaic conex swiching in blind signal separaion, IEEE Transacaions on Paern Analysis and Machine Inelligence, 22(10), (Ocober, 2000), [14] A. Hyvärinen, Fas and robus fixed-poin algorihms for independen componen analysis, IEEE Transacaions on Neural Newors, 10(3), (1999), [15] A. Hyvärinen, The fixed-poin algorihm and maximum lielihood esimaion for independen componen analysis, Neural Processing Leers, 10(1), (1999), 1-5.