TY - JOUR
T1 - A New Kind of Nonparametric Test for Statistical Comparison of Multiple Classifiers Over Multiple Datasets
AU - Yu, Zhiwen
AU - Wang, Zhiqiang
AU - You, Jane
AU - Zhang, Jun
AU - Liu, Jiming
AU - Wong, Hau San
AU - Han, Guoqiang
N1 - This work was supported in part by the NSFC under Grant 61332002, Grant 61300044, Grant 61472145, Grant 61572199, Grant 61502174, and Grant 61502173, in part by the Guangdong Natural Science Funds for Distinguished Young Scholars under Grant S2013050014677, in part by the Fundamental Research Funds for the Central Universities under Grant D2153950, Grant 2014G0007, and Grant 2015PT016, in part by the Science and Technology Planning Project of Guangdong Province, China, under Grant 2015A050502011, Grant 2016B090918042, Grant 2016A050503015, and Grant 2016B010127003, in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Grant CityU 11300715, in part by the Hong Kong General Research under Grant (152202/14E), and in part by the Hong Kong Polytechnic University under Grant G-YM05 and Grant G-YN39.
Publisher Copyright:
© 2016 IEEE
PY - 2017/12
Y1 - 2017/12
N2 - Nonparametric statistical analysis, such as the Friedman test (FT), is gaining more and more attention due to its useful applications in a lot of experimental studies. However, traditional FT for the comparison of multiple learning algorithms on different datasets adopts the naive ranking approach. The ranking is based on the average accuracy values obtained by the set of learning algorithms on the datasets, which neither considers the differences of the results obtained by the learning algorithms on each dataset nor takes into account the performance of the learning algorithms in each run. In this paper, we will first propose three kinds of ranking approaches, which are the weighted ranking approach, the global ranking approach (GRA), and the weighted GRA. Then, a theoretical analysis is performed to explore the properties of the proposed ranking approaches. Next, a set of the modified FTs based on the proposed ranking approaches are designed for the comparison of the learning algorithms. Finally, the modified FTs are evaluated through six classifier ensemble approaches on 34 real-world datasets. The experiments show the effectiveness of the modified FTs.
AB - Nonparametric statistical analysis, such as the Friedman test (FT), is gaining more and more attention due to its useful applications in a lot of experimental studies. However, traditional FT for the comparison of multiple learning algorithms on different datasets adopts the naive ranking approach. The ranking is based on the average accuracy values obtained by the set of learning algorithms on the datasets, which neither considers the differences of the results obtained by the learning algorithms on each dataset nor takes into account the performance of the learning algorithms in each run. In this paper, we will first propose three kinds of ranking approaches, which are the weighted ranking approach, the global ranking approach (GRA), and the weighted GRA. Then, a theoretical analysis is performed to explore the properties of the proposed ranking approaches. Next, a set of the modified FTs based on the proposed ranking approaches are designed for the comparison of the learning algorithms. Finally, the modified FTs are evaluated through six classifier ensemble approaches on 34 real-world datasets. The experiments show the effectiveness of the modified FTs.
KW - Classification
KW - classifier ensemble
KW - Friedman test (FT)
KW - nonparametric test
KW - statistical test
UR - http://www.scopus.com/inward/record.url?scp=85076286340&partnerID=8YFLogxK
UR - https://ieeexplore.ieee.org/document/7581018/authors#authors
U2 - 10.1109/TCYB.2016.2611020
DO - 10.1109/TCYB.2016.2611020
M3 - Journal article
C2 - 28113414
AN - SCOPUS:85076286340
SN - 2168-2267
VL - 47
SP - 4418
EP - 4431
JO - IEEE Transactions on Cybernetics
JF - IEEE Transactions on Cybernetics
IS - 12
ER -
