arxiv: v1 [cs.lg] 6 Mar 2018

Transcription

1 Deep Information Networks Giulio Franzese, Monica Vist Dipartimento di Elettronica e Telecomunicazioni Politecnico di Toro, Italy giulio.franzese@polito.it, monica.vist@polito.it arxiv: v1 [cs.lg] 6 Mar 2018 Abstract We describe a novel classifier with a tree structure, designed usg formation theory concepts. This Information Network is made of formation nodes, that compress the put data, and multiplexers, that connect two or more put nodes to an put node. Each formation node is traed, dependently of the others, to mimize a local cost function that mimizes the mutual formation between its put and put with the constrat of keepg a given mutual formation between its put and the target (formation bottleneck). We show that the system is able to provide good results terms of accuracy, while it shows many advantages terms of modularity and reduced complexity. Keywords: Information Bottleneck, Classifier, Neural Network, Information Theory. 1 Introduction The so-called formation bottleneck was described [1, 2], where it was employed to analyze how the formation flows side a classification neural network. We here propose to exploit the same idea to build a supervised learng mache that compresses the put data and generates the estimated class, usg a modular structure with a tree topology. The key element of the proposed Deep Information Network (DIN) is the formation node, that compresses the put while keepg its formation content: durg the trag phase the node evaluates the conditional probabilities P (X = j X = i) that mimize the mutual formation I(X ; X ) between its put X and put X for a given mutual formation between its put and the target I(X ; Y ). The put of each formation node at the first layer is one of the features/attributes of the data; once the conditional probabilities are found, the node randomly generates its put accordg to them. The second element of the DIN is the multiplexer that combes the puts of two or more formation nodes to generate the put of the subsequent one. The mache is therefore made of several layers, and the number of formation nodes per layer decreases from layer to layer, thanks to the presence of multiplexers, until a unique formation node is obtaed the last layer, whose task is to provide the estimated class of the put. Thus a DIN is similar to a neural network, but the workg prciple is completely different. In particular, there is no global objective function and each formation node is separately traed, usg only its put and the target class. The layers are traed order: the nodes of the first layer are traed usg the available raw data (one node for each of the available features) and stochastically generate the data for the multiplexers that feed the second layer; the nodes at the second layer are traed usg their put (ideally the formation contaed two or more features the raw data) and the target, etc. Each formation node has no 1

2 knowledge ab the put and put of other nodes. From layer to layer, the formation nodes and the multiplexers allow to extract the formation content of the origal data as if they were siftg it. Whereas algorithm C4.5 by Quland [3] builds a hierarchical decision tree startg from the root, we here propose a way to build a similar tree, but startg from the leaves. The advantages of DINs are clear: extreme flexibility, high modularity, complexity that learly depends on the number of nodes, overall number of nodes that learly depends on the number of features of the data. Sce the mache works with formation theory, categorical and missg data are easily managed; on the other hand, contuous data should be first quantized, but the true quantization can be left to the first layer formation nodes. What has to be assessed for the proposed mache is the performance: does it provide good results terms of misclassification probability? are there constrats on its parameters? Section 2 more precisely describes the structure of the DIN, section 3 gives some sight on the theoretical properties and section 4 comments the results obtaed with a dataset available on the web. Conclusions are fally drawn section 5. 2 The formation network As poted 1, the mache is made of formation nodes, described section 2.1, and multiplexers joed together through a network described section The formation node Each formation node, described Fig. 2.1, has an put vector x, whose elements take values a set of cardality N, and an put vector x, whose elements take values a set of cardality N ; moreover the target class vector y tra is available as put, with elements the set [0, N class 1]; all the vectors have the same size N tra. By settg N < N, the formation node performs a compression (source encodg). Let X (X, Y ) identify the random variable whose samples are stored vector x (x, y). In the trag phase, the formation node randomly generates the put vector x, accordg to the conditional probability mass function that satisfies equation [1] P (X = j X = i) = where 1 Z(i; β) P (X = j)e βd(i,j), i = 0,..., N 1, j = 0,..., N 1 (1) x N y x N Figure 1: Schematic representation of an formation node, showg the put and put vectors, with alphabets of cardality N and N, respectively, and the target vector y which is available durg the trag phase. 2

3 P (X = j) is the probability mass function of the put random variable X P (X = j) = 1 i=0 d(i, j) is the Kullback-Leibler divergence and d(i, j) = N class 1 m=0 P (X = i)p (X = j X = i), j = 0,..., N 1 (2) P (Y = m X = i) log 2 P (Y = m X = i) P (Y = m X = j) = KL(P (Y X = i) P (Y X = j)) (3) P (Y = m X = j) = β is a real positive parameter N 1 i=0 Z(i; β) is a normalizg coefficient that allows to get 1 j=1 P (Y = m X = i)p (X = i X = j), m = 0,..., N class 1, j = 0,..., N 1 (4) P (X = j X = i) = 1 (5) The probabilities P (X = j X = i) can be iteratively found usg the Blahut-Arimoto algorithm [1]. Accordg to [1], Eqn. (1) solves the formation bottleneck: it mimizes the mutual formation I(X ; X ) under the constrat of a given mutual formation I(Y ; X ). In particular, eqn. (1) is the solution of the mimization of the Lagrangian L = I(X ; X ) βi(y ; X ) (6) If the Lagrangian multiplier β is creased, then the constrat is privileged and the formation node tends to maximize the mutual formation between its put X and the class Y ; if β is reduced, then mimization of I(X ; X ) is obtaed, thus a compression is obtaed. The formation node must actually balance compression from X to X and propagation of the formation on Y, order to correctly decide its value. In our implementation, the compression is also imposed by the fact that the cardality of the put alphabet N is smaller than that of the put alphabet N. 2.2 The network Fig. 2 shows an example of a DIN, where we assume that the dataset is made of a matrix X with N rows and D = 8 columns (features) and the correspondg class vector y; we use N tra rows for the trag phase and N test = N N tra rows for the testg phase, gettg matrices X tra and X test (see Fig. 3). Now refer to Fig. 2. The k-th column x(k) of matrix X tra should be, together with vector y tra, the put of the formation node (k, 0) at layer 0 of the structure. However the algorithm needs a fite number of values the put vectors, and a pre-processg 3

4 N x q (0) x q (1) x q (2) x q (3) x q (4) x q (5) x q (6) x q (7) 0, 0 1, 0 3, 0 4, 0 5, 0 6, 0 7, 0 2, 0 Layer 0 x (0, 0) x (1, 0) x (0, 1) 0, 1 1, 1 2, 1 3, 1 x (0, 1) x (1, 1) Layer 1 x (0, 2) 0, 2 N (2) 1, 2 N (2) Layer 2 x (0, 2) x (1, 2) N (2) N (2) x t (0, 3) x (0, 3) 0, 3 N (3) N (3) = N classes Layer 3 Figure 2: Architecture of a simple formation node network for D = 8: each fo node is represented by a circle, the numbers side the circle identify the node, the triangles identify the mixers, N (k) (k) is the number of values taken by the put of the fo node at layer k, N is the number of values taken by the put of the fo node at layer k. X y Ntra Ntest x(0) x(1) Xtest Xtra x(d 1) ytest ytra D Figure 3: Defition of matrices, vectors and their dimensions 4

5 quantization phase is needed. Quantization (uniform, Max-Lloyd, etc.) is not discussed here, the example described Sect 4 we used lear quantization. Thus the true put of node (k, 0) is x q (k), the quantized version of x(k). Of course quantization is not necessary for categorical data. The number of nodes at layer 0 of the structure is D, equal to the number of columns of X. Information node (k, 0) at layer 0 thus processes x q (k) and y tra, and generates the put vector x (k, 0) with alphabet of cardality. Note that this put vector is randomly generated by the formation node, accordg to its probability matrix P k,0, whose element i, j is P (X = j X = i) as found by the algorithm described section 2.1. The put vectors x (2k, 0) and x (2k + 1, 0) are combed together by a multiplexer (shown as a triangle Fig. 2) that puts x (k, 1) = x (2k, 0) + x (2k + 1, 0) (7) Vector x (k, 1) is now the put of the formation node (k, 1) of layer 1, and has an alphabet with cardality = The topology is iterated halvg the number of formation nodes from layer to layer, until an isolated formation node is obtaed at layer d = log 2 (D), that has put vector x (0, d) with alphabet cardality N (d) and put vector x (0, d), whose alphabet cardality must be N (d) = N class, the number of classes the target vector y tra. If the parameters are correctly chosen, the put vector x (0, d) is equal to y tra, and the probability matrix P d,0 has just one value equal to 1 each of its rows. A tree topology is thus obtaed. In the overall, the topology proposed Fig. 2 requires a number of formation nodes equal to (8) N nodes = D + D 2 + D = 2D 1 (9) and a number of mixers equal to N mixers = D 2 + D = D 1 (10) All the nodes run exactly the same algorithm and all the mixers are equal, apart from the put/put vector alphabet cardalities. If the cardalities of the alphabets are all equal, i.e. N (i) (i) and N do not depend on the layer i, then all the nodes and all the mixers are exactly equal, which might help a possible hardware implementation. 2.3 The testg/runng phase Durg the testg or runng phase, the quantized columns of matrix X test are used as puts and each node simply generates the put vector x accordg to the put vector x, usg the probability matrix P that has been optimized durg the trag phase. For example, if the n-th value stored the vector x (k, l) of node k, l is i, then the n-th value vector x (k, l) is j with probability P k,l (i, j). Each node then generates a stochastic put, accordg to its probability matrix. 3 Analysis In the followg, the DIN is studied considerg first the overall conditional probabilities between the put features and the put, and second the formation flow side it. 5

6 X,a N0 N 0 X,b a b X,a N 1 N 1 X,b X,c N 1 N 1 c X,c N 2 Figure 4: Sub-network used for the evaluation of the probability matrix; X,a, X,a, X,b, X,b, X,c, X,c are all random variables; N 0 is the number of values taken by X,a and X,b, N 1 is the number of values taken by X,a and X,b, N 2 is the number of values taken by X,c. 3.1 The probabilistic pot of view Consider the case of the sub-network made of nodes a, b, c as shown Fig. 4; the alphabet cardality at the put of nodes a and b is N 0, the cardality at the put of nodes a and b is N 1, the alphabet cardality at the put of node c is N 1 N 1, the cardality at its put is N 2. Node a is characterized by matrix P a, whose element P a (i, j) is P (X,a = j X,a = i); similar defitions hold for P b and P c. Note that P a and P b have N 0 rows and N 1 columns, whereas P c has N 1 N 1 rows and N 2 columns; the overall probability matrix between the puts X,a, X,b and the put X,c is P with N 0 N 0 rows and N 2 columns. Then P (X,c = i X,a = j, X,b = k) = = = r=0 r=0 r=0 s=0 s=0 s=0 It can be shown that P (X,c = i, X,a = r, X,b = s X,a = j, X,b = k) P (X,c = i X,b = r, x,c = s)p (X,a = r X,a = j)p (X,b = s X,b = k) P (X,c = i X,b = r, X,c = s)p a (j, r)p b (k, s) (11) P = (P a P b )P c (12) where identifies the Kronecker matrix multiplication; note that P a P b has N 0 N 0 rows and N 1 N 1 columns. By iteratively applyg the above rule, we can get the expression of the 6

7 overall matrix P for the exact topology of Fig. 2, with 8 put nodes and four layers: [ ] [ ] P = {[(P0,0 } P 1,0)P 0,1 (P2,0 P 3,0 )P 1,1 P 0,2 { [(P4,0 ] [ ] } ] P 5,0 )P 2,1 (P6,0 P 7,0 )P 3,1 P 1,2 P 0,3 (13) The DIN then behaves like a one-layer system that generates the put accordg to matrix P, whose size might be impractically large (with D features all quantized with N 0 levels, matrix P has size N0 D 2). The proposed layered structure needs smaller probability matrices, which makes the system computationally efficient. 3.2 The formation theory pot of view Each formation node compresses the data and therefore, by a basic application of data processg equality, we have I(X ; Y ) I(X ; Y ) (14) Consider now the puts X (2k, i) and X (2k + 1, i) of two nodes at layer i that are combed by the mixer to generate X (k, i + 1). It is terestg for analysis and tegrity purposes to derive an upper and a lower bound of the total amount of formation obtaed at the put of the mixer. First of all notice that, from an formation theoretic pot of view, we can write I(X (k, i + 1); Y ) = I([X (2k, i), X (2k + 1, i)]; Y ) (15) We can expand (15) as I([X (2k, i), X (2k + 1, i)]; Y ) = H([X (2k, i), X (2k + 1, i)]) H([X (2k, i), X (2k + 1, i)] Y ) = H(X (2k + 1, i) X (2k, i)) + H(X (2k, i)) (H(X (2k + 1, i) X (2k, i), Y ) H(X (2k, i)) Y ) = I(X (2k, i); Y ) + I(X (2k + 1, i) X (2k, i); Y ) (16) = I(X (2k + 1, i); Y ) + I(X (2k, i) X (2k + 1, i); Y ) (17) where the last equality is due to evident symmetries the equations. Combg (16) and (17), and due to non-negativity of mutual formation, we obta I(X (k, i + 1); Y ) max{i(x (2k, i); Y ), I(X (2k + 1, i); Y )} (18) where equality is attaed when H(X (2k + 1, i) X (2k, i)) = 0 (i.e. when X (2k + 1, i) can be exactly predicted if X (2k, i) is known). We can also derive an upper bound for the total mutual formation. First we observe that H([X (2k, i), X (2k + 1, i)]) H(X (2k, i)) + H(X (2k + 1, i)) (19) H([X (2k, i), X (2k + 1, i)] Y ) max{h(x (2k, i) Y ), H(X (2k + 1, i) Y )} (20) and thus we can derive, usg (19) and (20), that I(X (k, i + 1); Y ) = H([X (2k, i), X (2k + 1, i)]) H([X (2k, i), X (2k + 1, i)] Y ) H([X (2k, i), X (2k + 1, i)]) H(X (2k, i) Y ) H(X (2k, i)) + H(X (2k + 1, i)) H(X (2k, i) Y ) = I(X (2k, i); Y ) + H(X (2k + 1, i)) (21) = I(X (2k + 1, i); Y ) + H(X (2k, i)) (22) 7

8 where aga the last equality is due to symmetry. We can then combe (21) and (22) I(X (k, i + 1); Y ) m{i(x (2k, i); Y ) + H(X (2k + 1, i)), I(X (2k + 1, i); Y ) + H(X (2k, i))} (23) Fally, combg (18) and (23) max{i(x (2k, i); Y ), I(X (2k + 1, i); Y )} I(X (k, i + 1); Y ) m{i(x (2k, i); Y ) + H(X (2k + 1, i)), I(X (2k + 1, i); Y ) + H(X (2k, i))} (24) obtag an upper and lower bound for the mutual formation. 4 The Kidney Disease experiment In this section we present the results of an experiment performed on the UCI Kidney Disease dataset [5, 6]. The dataset has a total of 24 medical features, consistg of mixed categorical, teger and real values, with missg values present. The aim of the experiment is to correctly classify patients affected by chronic kidney disease. Remember that the mache works also with missg values, sce they are seen a a special categorical value, whose probability is the only required measure. In the proposed formation network, layer zero has as many formation nodes as the number of features (i.e. 24), and the put of each node is one of the quantized features; these nodes are traed parallel. Then the puts of layer zero are mixed two at a time and they become the put of layer one with 12 formation nodes. These 12 nodes are merged to 6 nodes to layer two and the 6 nodes are mixed to 3 nodes layer three. Fally the three nodes are combed to the unique fal node whose put cardality is equal to 2 (correspondg to ill or healthy). The 24 put features are uniformly quantized with different values of, dependg on the feature (from 2 to 470); on the contrary the value of N is 3 for all the nodes apart from the last one. The multiplexers generate vectors takg values the range [0, 8], and the formation nodes compress these values the range [0, 2]. The value of β is equal to 5; the number of trag pots is equal to N tra = 200 (half of which positive), the number of testg pots is aga N test = 200. Figure 5 shows how the mutual formation I(X; Y ) evolves the DIN, and it is possible to appreciate that it creases from the first to the last layer. In particular I(X (2k, i); Y ) and I(X (2k + 1, i); Y ), represented as circles, are lked through segments to I(X (k, i + 1); Y ), represented as a triangle, to show that the lower bound (24) holds from i = 0 (blue markers), to i = 3 (black markers). Figure 5 also shows that I(X (k, 1); Y ) (blue triangle) is higher than I(X (k, 1); Y ) (green circle at the same x-value), which is due to the compression performed by the formation node, that reduces the cardality of X (k, 1) to just three values; this is true general when I(X (k, i); Y ) is compared to I(X (k, i); Y ) for all values of k and i. In terms of performance, the results listed table 1 were obtaed: some form of overfittg is present when N (i) = 3 and better results are obtaed when N (i) = 2, i = 1,..., 3 accuracy, specificity and sensitivity are high all the considered cases when N (i) = 2, the accuracy obtaed with N tra = 200 is similar to that obtaed with N test = 320, which shows that 200 patients are enough to tra the DIN 8

9 Figure 5: Mutual formation the network: circles correspond to puts of the multiplexers, triangles to their puts, each dotted le represents the maximum value of the mutual formation at the put of the multixers of a given layer. Results obtaed durg one run of the trag phase with N tra = 200 (half ill, half healthy), N (i) = 3. N (i) N tra N test Accuracy Sensitivity Specificity F1 score trag testg trag testg trag testg trag testg Table 1: Results obtaed with DINs on the Kidney Disease dataset; averages over 1000 runs; β = 5, the specified value of N (i) (4) is valid for i = 0, 1, 2, 3, the last layer has N = N classes = 2. 9

10 results obtaed with N tra = 320 and N test = 80 can be comparable with those obtaed [6] usg other algorithms (k-nearest neighbour, random forest, neural networks) Notice that the DINs were run one thousand times, each time randomly changg the subsets of patients used for the trag and the testg phases; table 1 shows the average results. 5 Conclusions The proposed Deep Information Network (DIN) shows good results terms of accuracy and represents a new simple modular structure, flexible and useful for various applications. Further optimization might be obtaed by usg a different value of N for each of the formation nodes (not necessarily all equal at each layer) and appropriately selectg the value of the Lagrangian multiplier β. References [1] N. Tishby, N. Zaslavsky, Deep Learng and the Information Bottleneck Prciple arxiv: v1 [cs.lg], March 2015 [2] N. Tishby, F. C. Pereira, W. Bialek The formation bottleneck method arxiv:physics/ v1, April 2000 [3] J. R. Qulan, C4.5: Programs for Mache Learng. Morgan Kaufmann Publishers, 1993 [4] M. Cover, J. A. Thomas, Elements of Information Theory, 2nd Edition (Wiley Series Telecommunications and Signal Processg), 2006 [5] M. Lichman, UCI Mache Learng Repository, University of California, Irve, School of Information and Computer Sciences, [6] A. Salek, J. Stankovic, Detection of Chronic Kidney Disease and Selectg Important Predictive Attributes, 2016 IEEE International Conference on Healthcare Informatics (ICHI), 4-7 Oct. 2016, DOI: /ICHI