PERFORMANCE COMPARISON OF THREE DATA MINING MODELS FOR BUSINESS TAX AUDIT

Transcription

1 PERFORMANCE COMPARISON OF THREE DATA MINING MODELS FOR BUSINESS TAX AUDIT 1 TSUNG-NAN CHOU 1 Asstt Prof., Department of Finance, Chaoyang University of Technology. Taiwan 1 tnchou@cyut.edu.tw ABSTRACT Although the taxation audit can increase the government revenue and motivate taxpayers to comply with tax laws, the audit processes could be labor and time intensive and require the information technology to make the audit programs more effective. The data mining models including logistic regression, artificial neural network and rough set analysis were examined in this study for the selection of tax audit cases, and their performances were compared with various metrics. The experimental results indicated the rough set analysis performed better in most evaluation metrics and its accuracy could be significantly improved to 74 % with a sensitivity of 70 % and specificity of 77 %, respectively. In addition to the accuracy measure, other metrics such as Precision, Youden Index, Kappa, Odds Ratio, Likelihood Ratios and statistic tests also suggested that rough set model achieved the best performance, and could be effectively applied to select the audit candidates of business tax. Keywords: Artificial Neural Network, Logistic Regression, Rough Set, Tax Audit 1. INTRODUCTION The economic growth of a country is profoundly affected by the fiscal policy tools such as taxation and government spending. The taxation, in particular, is considered as the primary government revenue which collected to support and finance government expenditures. Mostly, the government needs to look at ways of keeping a balance between taxation and government spending since changing the balance will incur budget deficit and surplus. However, it is not easy to increase the government expenditures without raising taxes to fund the government. Moreover, increasing taxes might drive businesses change their investment and tax behaviors in order to reduce the tax expenditure and result in serious economic problems. Alternatively, instead of raising taxes, another solution to increase revenue for government is to focus on reducing the amount of tax evasion and fraud such as underpayment of tax, underreporting of income, misrepresenting deductions and offshore tax evasion. According to the statistics reports from the Taxation Administration of Ministry of Finance in Taiwan [1], the total net tax revenue increased from NT$ 1,601 billion to NT$ 1,834 billion over the last eight years, though it fell 8 slightly in 009 and 010 as shown in Table 1. In November 014, the total net tax revenue was reported as NT$ 187 billion, which increased by 9.9 % more than the same month last year. The cumulated amount from January to November in 014 were NT$ 1,854 billion, which was 8. % more than the same period in previous year. On the other hand, the Table 1 also illustrated the growth of the three major taxes from 006 to 013. For tax year 013, the government collected NT$ 1,834 billion in tax revenue, over half of which came from those taxes. The Individual Income Tax represented for NT$ 39 billion, or 19 % of all taxes. The revenue of Business Tax totaled NT$ 303 billion and accounted for 17 %, while Profit-seeking Enterprise Income Tax plunged to NT$ 86 billion in 010 and then quickly grew to NT$ 351 billion, which represented for 1 % of all taxes. In spite of the rising revenue of government, the number of suspected cases of tax evasion selected for audit dropped from 74,400 to 08,760 over the past eight years as shown in Table. In 008 there was a substantial growth in fined cases, and in the following years the total growth of fined cases went down to about 171,587 of 013. On the other

2 hand, the fined amount has been significantly dropped from NT$ 5,770 million to NT$ 8,358 million, while average Audit Failure Rate increased from 11% to 16%. Indeed, Audit Failure Rate for Profit-seeking Enterprise Income Tax, Individual Income Taxes and Business Tax were 5%, 3% and 54% in 013, respectively. Obviously, the failure rate of Business Tax was significantly higher than others and raised a serious audit challenge for tax inspectors. On the whole, the taxation audit process is intended to deal with the issues related to payment of tax obligations and taxpayer reporting, such as underpayment of tax and underreporting of income. To improve the quality of performing tax audit will not only increase the government revenue but also motivate taxpayers to comply with tax laws and ensure proper reporting of taxable events and liabilities. However, tax audits could be labor intensive processes and require the information technology such as machine learning techniques to make the audit program more effective. Basically, the data mining models might provide considerable solutions for tax agencies to extract information from the various data sources and identify appropriate candidates for tax audit and comply with the selection criteria of audit policy. In this paper, we focus on the tax evasion in Business or Corporate Tax only since it accounted for over 0% of all tax revenues, and achieved the highest Audit Failure Rate than other taxes. The next section will briefly outlines the applied data mining models. After an overview of evaluation metrics, the experimental results will be given in details. The final section provides a conclusion of the work and gives a brief summary of the findings.. DATA MINING MODELS Normally, to audit all taxpayers with the limited time and labors is not applicable for tax agencies. Moreover, the overall cost associated with tax audits might exceed the revenue they generated. As a result, machine learning and statistical approaches were used to improve the audit quality by identifying potential fraud cases in business tax because it accounted for over 0% of all tax revenues and performed worse than others in Audit Failure Rate. The data mining models including logistic regression, artificial neural network and rough set analysis were applied to assist audit selection and compare their performances with various metrics. Logistic regression is a probabilistic statistical classification model for analyzing a dataset in which the dependent variable is discrete rather than continuous, and could be encoded as 1 or 0. The logistic regression in this study attempts to find the best fitting model to describe the relationship between the dependent variable which represents by tax fraud and no tax fraud and a set of independent variables which includes the various combination of the financial statement and annual business income tax report. This approach generates the coefficients accompanied with its standard errors and significance levels based on maximizing the likelihood of observing the sample values []. The probabilities describing the possible outcomes of dependent variable have two categories: fraud and no fraud. The following equation illustrates one of the feasible fitting solution of this study, and the logistic function, Logit(p), is a link function, used to transform the probability of the dichotomous variable into a continuous variable. Logit(p) = *[X]-0.3*+[X9] The second model evaluated in this study is a feed forward artificial neural network model based on back propagation learning algorithm [3]-[4]. The model maps a set of independent variables onto a set of appropriate outputs which represent the decision variable or dependent variable. The mapping process is fulfilled through a network structure that consists of multiple layers of neural nodes with a nonlinear activation function for each. In contrast to the logistic regression model, the output of neural network is a nonlinear function of independent variables. In addition, the sigmoid function applied as activation function for neural network is a hyperbolic tangent which ranges from -1 to 1, while the logistic function used in logistic regression has similar S curve shape but ranges from 0 to 1. The hidden layer of the neural network is defined by (Number of independent variables + Number of dependent variables outcomes) /. For speeding up the convergence of the training process, the learning rate and momentum are set to 0.5 and 0. respectively. The input variables are also normalized to improve performance of the network. 9

3 The third model employs the rough set model proposed by Pawlak in [5]-[6], as the rough set analysis is useful for reasoning and classifying about categorical data. The collected data is organized as a decision table which composed of condition variables and decision variables. Unlike the statistical approaches, rough set inference does not need any preliminary probability distribution about data, and this approach extracts sets of decision rules directly from simplified sets of data which possesses the same information as its original data. To achieve better performance, both the discretization and linear combination for numerical data addressed in [7] are employed to generate more effective inference rules. The former discretizes the variables in the data table by a set of calculated cuts according to prespecified discretization algorithm. For instance, the continuous variable X1 is transformed to a nominal variable with three distinct values according the specified cuts in [ , , ]. On the other hand, the latter creates a weighted sum of selected variables through a linear equation created by the adaptive optimization algorithm. Totally, there are four linear equations such as X3*0.13+X4*0.976+X6*0.016, were used to create new variables and improve the classification accuracy in this study. 3. PERFORMANCE METRICS To understand the goodness of a model with respect to its ability to discriminate between nofraud and fraud cases, as well as performing quantitative comparison for different classification models, several metrics are chosen to determine the efficacy and accuracy of algorithms in this study. As the detection of tax fraud is a binary classification tasks, the performance of a model is measured by two different criterions including confusion matrix based metrics and hypothesis testing. The former is to compare the number of predicted frauds (no-frauds) to the actual number of frauds (no-frauds) and represent this outcome as a confusion matrix. A number of common metrics are therefore derived from the matrix. The latter gives a null hypothesis that all models perform the same and compute test statistic to accept or reject the hypothesis according to selected testing methods. Both criterions are described below. 3.1 Confusion Matrix Based Metrics The confusion matrix contains information about actual and classified outcomes measured by a model, and can be used to evaluate the quality of classification. The classified result is organized as a four folder matrix with the predicted outcomes shown in the rows and the actual outcomes shown in the columns respectively. As illustrated in Table 3, each cells in the table indicates the number of True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN) respectively. Actual Table 3. Confusion Matrix Classification Models Predicted No-Fraud Fraud No-Fraud TP FP Fraud FN TN Based on the consolidation and variation of the four basic measures, a number of common metrics can be derived from the confusion matrix to assess the performance of models. As listed in Table 4, there are totally 10 metrics, which addressed in [8]- [10] are applied to examine the quality of classification models. Table 4. Confusion Matrix Based Metrics Sensitivity = TP / (TP + FN) Specificity = TN / (TN + FP) Precision = TP / (TP + FP) Accuracy = (TP +TN)/ (TP + FP + FN +TN) False Positive Rate = FP / (FP + TN) False Negative Rate = FN / (TP + FN) Kappa = (Total Accuracy-Random Accuracy) / (1- Random Accuracy) Youden Index = Sensitivity (1 Specificity) Likelihood Ratios: LR(+)=Sensitivity / (1- Specificity) LR(-)=(1 Sensitivity) / Specificity Odds Ratio=( TP / FN) / (FP / TN)= LR(+) / LR(-) Normally, some trade-off between the Type I and Type II errors need to be take into account prior to implementation since it is unlikely to achieve a perfect model with 100% sensitivity and specificity in classifying all the fraud and no-fraud cases correctly. A false positive represents a Type I error and means that a tax agency incorrectly audit a potentially good individual or business. On the other hand, a false negative corresponding to a Type II error is more crucial as the tax inspectors fail to identify a potentially individual or business with tax evasion. 10

4 Avoiding the type I errors that classify no-fraud case as fraud case could prevent a waste of time from the unsuccessful prosecutions thereafter. Nevertheless, if the top priority of the tax authority in the task is to uncover the fraud cases instead of cost saving, then reducing the type II errors will be the most concerned issue. Under any circumstances, the tax authority needs to balance both types of errors in order to reduce the overall losses that generate revenues less than the cost and efforts associated with the audit process 3. Hypothesis Testing Metrics In addition to select the best model based on the out-of-sample classified accuracy from confusion matrix, some statistical criterions are also required to perform multiple comparisons among the three presented models. The McNemar test can be used to investigate whether one model is significantly better than the other based on the matched pairs of testing data. As shown in Table 5, the McNemar test is a non-parametric test arranges the outcomes of two models in contingency table [11]-[1]. Table 5. Contingency Table ( McNemar Test) Model B Model A No-Fraud Fraud No-Fraud S 00 S 01 Fraud S 10 S 11 Both the parameters S 01 : and S 10 represent the number of observed dissimilarities and can be used to indicate how the performances differ between models. The null hypothesis H 0 : S 01 = S 10 assumes that different classifiers perform similarly. If the calculated χ value with 1 degree of freedom is larger than 3.84 at a 95% confidence test, then the null hypothesis is rejected and indicate the models performs differently. S10 S01 1 ( S S ) Another non-parametric Cochran's Q test [1] is applied to measure whether there are significant differences or heterogeneity between a numbers of models on the same testing data. This test can be regarded as an extension of the McNemar test but incorporates more models for comparison. The null hypothesis for the Cochran's Q test is that there are no differences between the classification accuracy of models, H 0 : MA= MB= MC. If the Cochran's Q test is positive, in other word, the calculated Q value is larger than the Chi-Squared value corresponding to the selected significance level, the 11 null-hypothesis is rejected and suggested that the three models are significantly different from each other. The Cochran's Q is given by the following with the degrees of freedom k-1. Q k( k 1) k k k G j ( k 1)( j 1 j 1 b b k Li Li i 1 i 1 G ) k is the number of compared models G j is the column total for the j th model b is the number of instances in each model L i is the row total for the i th instance 4. EMPIRICAL STUDY AND RESULTS In this section, the experimental results were presented for three compared models. The first model was a neural network classifier and denoted as NNW. In addition, the second model denoted as LOGIT was a simple logistic regression analysis and the third model denoted as RS was a rough set model with a specified discretization of variables which transformed continuous data into discrete counterparts. In order to examining the quality of the classification models, the suspected cases of tax evasion collected from a tax agency were imported and separated as the training and testing data. The data sets consisted of the financial statement and tax report of the suspected corporate tax evader with a mix of categorical and continuous variables, and the dependent variable gave a value of 0 if there was no fraud and a value of 1 if there was a fraud. The data containing 10 independent variables were segmented into fraud cases and no fraud cases, and then were randomized to avoid sampling bias. To provide sufficient and adequate data for the evaluation of models, a total of 33 cases were reviewed and subdivided into 3 for the training dataset and another 100 for the testing dataset. To deal with the data of different units and scale, both the training and testing datasets were normalized and converted to a common scale prior to the classification analysis. The first part of model comparison focused on comparing their respective confusion matrices. For simplicity, several metrics described in section 3 were selected for performance evaluation instead of using the elements of confusion matrix directly. A major benefit of using these metrics was j

5 transforming the comparison of all models into a single value metric. The sensitivity and specificity are two measures that commonly used to evaluate the performance of models. The sensitivity accounts for the probability that the classified outcome will be tax fraud among those who are in fact tax fraud. In contrast, the specificity indicates the probability that classified outcome will be no fraud among those who are actually no fraud. On the other hand, the false positive rate is equal to 1-specificity and the false negative rate is equal to 1-sensitivity. In essence, the sensitivity and specificity capture the same information as the false positive and false negative rates. These metrics are interchangeable, but often times, sensitivity and the false positive fraction are reported together for model evaluation. The results of these metrics were summarized below. By referring to Table 6, the Specificity of RS model was better than both NNW and LOGIT models. The result showed that the True Negatives (fraud case) were correctly classified by RS model with a probability of 77% and misclassified with a probability of 3%. However, the Sensitivity of RS model was 70%, and indicated that it was slightly weaker in identifying the true positives correctly if compared to NNW model. As a highlight, considering NNW and RS models, NNW model achieved higher Sensitivity, while RS model had higher Specificity. It was not easy to achieve an acceptable balance between specificity and sensitivity. If the fined amount of tax fraud case was a major issue concerning the revenue of government in terms of false negatives, then Specificity was very important. As a result, RS model was the best choice. On the other hand, if the tax agency cared about the waste of time and efforts on identifying no fraud cases, then the Sensitivity was important and NNW model could be the favorable model. Basically, chose the RS model could gain more specificity, by losing a bit of sensitivity. The performance measures of the Precision, Accuracy and Youden Index were given in Table 7. Based on the traditional accuracy measure, the RS model was the best model that reached the highest Accuracy at 74%. Despite the fact that the accuracy was vulnerable to the unbalanced dataset, the accuracy measure could still be used to assess the overall effectiveness of models. When comparing models with Precision measure, the RS model was better than NNW and LOGIT models by as large as 15%, increased from 53% to 70%. Besides, the Youden Index was a measure derived from the sum of Sensitivity and Specificity and was not affected by the prevalence of specific class within an unbalance dataset. Therefore, the Youden Index could be used to evaluate the overall discriminative power of a model even though it was not sensitive for differentiating the sensitivity and specificity. As shown in Table 7, the RS model reached the highest Youden Index at 0.47 and was a good discriminant when compared to the other two models. Naturally the Sensitivity and Precision measures as well as their combinations focus only on the True Positive cases rather than handles True Negative cases. To assess the overall performance of models, Kappa statistic was used to provide further comparison of models. For an unbalanced data, a model that simply output the more prevalent class could achieve a higher predictive accuracy. In that case, kappa value might give a clear measure about the validity of model while its precision or sensitivity was high. The Kappa test of +1 meant that the model was perfectly reliable. In Table 8, both NNW and LOGIT models obtained deteriorated Kappa values of 0.5 and 0.1 respectively. Despite the fact that the RS model performed a little lower value of 0.47, which worse than a perfect model, the RS model was nearly 0% better than NNW and LOGIT models. As the Table 8 showed, in the case of the RS model, the DLR(+) was 3.06 and DLR(-) was Since the higher positive likelihood ratio and a lower negative likelihood ratio represented better performance on positive and negative classes respectively. Both likelihood ratio metrics of RS model outperformed other two competing models and the evaluation result clearly favored the RS model. Another metric, the Odds Ratio was the positive likelihood ratio divided by the negative likelihood ratio, and higher Odd Ratio indicated better discriminative ability. As could be seen from Table 8, Odds Ratio was 7.81 for the RS model which was obviously much higher than other models. The result gave an impression that RS model was superior to the NNW and LOGIT models. The overall results of performance evaluation above demonstrated the RS model was an applicable model even the NNW model obtained higher Sensitivity. 1

6 Although the above metrics provided most basic measures for understanding the performance of a classification model, it was considered that hypothesis testing would usefully supplement the performance analysis. To demonstrate that the RS model was significantly better than the others on the same testing data, both Cochran's Q and McNemar tests were chosen to allow a further evaluation of the models. Cochran's Q test was a non-parametric statistical test proposed for measuring whether a number of classifiers had identical effects on the same testing data which represented by a value of either 0 or 1. The hypothesis for no difference between the accuracies of models was examined. If Cochran's Q test was found statistically significant, then the McNemar test was needed as post-hoc analysis to find where actually this difference was. The obtained Cochrane's Q value of 6.8 was larger than the critical chi-square value of 5.99 for degrees of freedom at α = 0.05 (Q() = 6.8, p <.05). The null hypothesis was rejected and indicated that there were significant differences among the models. As a result, the McNemar test was required to perform another test for pairwise comparison of models and found out which pair of models were significantly different. The null hypothesis of McNemar test assumed that there was no difference between the accuracies of two correlated models. Because the calculated chi-squared statistic was 6.55 (> 3.841) and corresponding p-value was , the null hypothesis was rejected and revealed that RS and LOGIT models were significantly different. On the other hand, for the RS and NNW models, the calculated chi-squared statistic was 4.83 and corresponding p-value was less than 0.05, supporting significant differences between both models. In addition, McNemar test also suggested there were no significant differences between NNW and LOGIT models, and the results were shown in Table CONCLUSION In this paper, three different data mining models were proposed for the tax inspectors to assist and automate their audit processes of tax evasion. The RS model was found significant improvement in performance with the weighted discretization and linear combination of data. According to the upon observations, the RS model outperformed other models since the experimental results showed that 13 the overall accuracy of the RS model could be improved to 74% with a sensitivity of 70% and a specificity of 77%, respectively. In addition to the accuracy measure, other metrics such as Precision, Youden Index, Kappa, Odds Ratio and Likelihood Ratios were also evaluated. All of these metrics indicated that RS model achieved the best performance and could be effectively used for classifying suspected fraud cases. Moreover, both the Cochran's Q test and the McNemar test of posthoc analysis also supported the significant differences between the RS model and other models. Following an overall approach of model evaluation, the rough set model achieved highest accuracy rate, and was sustained with other measures despite obtaining lower Sensitivity. With respect to overall effectiveness it could be concluded that rough set model performed better than other models in audit candidate detection and was able to efficiently reduce the audit failure rate to comply with the audit planning of taxation agencies. REFRENCES: 1. Growth of Tax Revenues and Total Net Tax Revenue (by Agency) as of August 014, Taxation Administration of Ministry of Finance, R.O.C. Sumner, M., Frank, E. and Hall, M.A. (005). Speeding up Logistic Model Tree Induction. In: 9th European Conference on Principles and Practice of Knowledge Discovery in Databases, , Taha, R.O.H. (01). The possibility of using artificial neural networks in auditing-theoretical analyticalpaper, European J. Economics, Finance and Administrative Sciences, 47, Darabi, Roya, Zohoriyan, Abolfazl and Etebar Shukufe (01). Declaration Tax Audit Using Chaos Theory (Neural Networks) Archives Des Sciences, Vol. 65, No Pawlak, Z. (198). Rough Sets, International Journal of Computer and Information Sciences, Vol.11, Pawlak, Z. (1991). Rough Sets - Theoretical Aspect of Reasoning about Data, Kluwer Academic Pubilishers. 7. Slezak, D., Wroblewski J. (1999). Classification algorithms based on linear combinations of features, in Proceedings of PKDD'99, LNAI 1704, springer- Verlag,

7 8. Stein, Roger M. (007). Benchmarking Default Prediction Models: Pitfalls and Remedies in Model Validation, Journal of Risk Model Validation, Vol. 1, No. 1, Subashini, T.S., Ramalingam, V. and Palanivel, S. (009). Breast mass classification based on cytological patterns using RBFNN and SVM, Expert Systems with Applications 36, Bekkar, M., Djemaa, H. K. and Alitouche, T. K. (013). Evaluation Measures for Models Assessment over Imbalanced Data Sets, Journal of Information Engineering and Applications, Vol.3, No Ludmila I. Kuncheva(014). Combining Pattern Classifiers: Methods and Algorithms, nd Edition, Wiley, Sheskin, D. J. (004). Handbook of parametric and nonparametric statistical procedures. 3rd ed. Boca Raton: Chapman & Hall. Table 1. Government Tax Revenues Profit-seeking Enterprise Tax Year Grand Total Individual Income Tax Business Tax Income Tax 006 1, , , , , , , , Unit:NT$billion Table. Tax Omission and Punitive Fines Tax Year Suspected Cases Fined Cases Fined Amount Audit Failure Rate ,400 36,76 5,770 11% ,561 65,37 5,010 11% ,907 88,08 5,113 11% ,598 83,317 31,479 15% ,579 68,556 1,549 17% ,890 11,495 1,675 14% 01 18, ,36 10,560 16% , ,587 8,358 16% 14

8 Table 6. Performance Measures (Sensitivity, Specificity, FP, FN) Model Sensitivity Specificity False positive False negative NNW LOGIT RS Table 7. Performance Measures (Precision, Accuracy, Youden Index) Model Precision Accuracy Youden Index NNW LOGIT RS Table 8. Performance Measures (Kappa, Odds Ratio, Likelihood Ratios) Model Kappa Odds Ratio DLR(+) DLR(-) A B C Table 9. Result of McNemar Test RS and LOGIT RS and NNW LOGIT and NNW N χ Statistic Exact Sig * * *p<.05 15