Detecting Changes in Offline and Online Classification Tasks

In machine learning, an essential assumption to build a well-performing classification model is that it should be trained and tested against data that come from the same distribution. However, in the real-world, once a model is in the deployment stage, the control over incoming data is limited. Accurately and efficiently detecting changes violating the fundamental assumption for classification tasks is crucial to ensure the reliability and performance of the artificial intelligence systems.

Different types of changes can arise in offline and online classification tasks. The goals and methods for change detection in the two scenarios are also different. As a starting point, this thesis first focuses on the detection of out-of-distribution examples in the testing data set in offline classification tasks. A purely unsupervised detector Label-Assisted Memory Auto-Encoder (LAMAE), and its refined version LAMAE+, are proposed to improve the detection of a wider range of out-of-distribution examples. Afterwards, this thesis progresses to the online classification scenario. In a streaming data environment, concept drift, which is a change in the underlying data distribution may occur. Instead of detecting single examples as in the offline scenario, online scenario requires sophisticated algorithms to identify if and when a change occurs in the underlying data distribution. This thesis proposes a novel concept drift detection framework named Hierarchical Reduced-space Drift Detection framework (HRDD) to meet this goal. HRDD not only recognizes a wider range of drifts regardless of their effects on classification performance, but also does so with an improved efficiency than existing methods. Another challenge faced by existing concept drift detectors is the assumption of data independence on data streams. To further approximate the reality, this thesis also

i attempts to investigate the new challenges brought by the relaxation of the independence assumption. A novel problem formulation is constructed taking into account temporal dependency, under which a greater variety of drift forms can possibly emerge. Afterwards, a simple and effective solution named Concept Drift detection for Temporally Dependent data streams (CDTD) to detect drifts, especially the ones that are being neglected by existing detectors, is presented.

In summary, this thesis tackles the detection of change in offline and online classification tasks. The approaches taken in the thesis are both efficient and effective, and have important significance in minimizing the disparity between the simulated environment and the physical reality.

[Abdelzad, Vahdat et al. (2019). “Detecting out-of-distribution inputs in deep neural networks using an early-layer output”. In: arXiv preprint arXiv:1910.10307.Abdi, Hervé and Lynne J Williams (2010). “Principal component analysis”. In: Wiley interdisciplinary reviews: computational statistics 2.4, pp. 433–459.Alippi, C, G Boracchi, and M Roveri (2016). “Hierarchical change-detection tests”. In:IEEE Transactions on Neural Networks & Learning Systems 28.2, pp. 246–258. Alippi, Cesare and Manuel Roveri (2008). “Just-in-time adaptive classifiers?Part I: Detecting nonstationary changes”. In: IEEE Transactions on Neural Networks 19.7, pp. 1145–1153.Alippi, Cesare, Giacomo Boracchi, and Manuel Roveri (2010a). “Adaptive classifiers with ICI-based adaptive knowledge base management”. In: International Conference on Artificial Neural Networks. Springer, pp. 458–467.— (2010b). “Change detection tests using the ICI rule”. In: The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–7.— (2011a). “A hierarchical, nonparametric, sequential change-detection test”. In: The 2011 International Joint Conference on Neural Networks. IEEE, pp. 2889–2896. — (2011b). “A just-in-time adaptive classification system based on the intersection of confidence intervals rule”. In: Neural Networks 24.8, pp. 791–800.— (2013). “Just-in-time classifiers for recurrent concepts”. In: IEEE transactions on neural networks and learning systems 24.4, pp. 620–634.— (2017). “Hierarchical change-detection tests”. In: IEEE Transactions on Neural Networks and Learning Systems 28.2, pp. 246–258.Aminikhanghahi, Samaneh and Diane J. Cook (2017). “A survey of methods for time series change point detection”. In: Knowledge and Information Systems 51.2, pp. 339367.An, Jinwon and Sungzoon Cho (2015). “Variational autoencoder based anomaly detection using reconstruction probability”. In: Special Lecture on IE 2.1, pp. 1–18.Andrews, Jerone TA, Edward J Morton, and Lewis D Griffin (2016). “Detecting anomalous data using auto-encoders”. In: International Journal of Machine Learning and Computing 6.1, p. 21. Antwi, Daniel K, Herna L Viktor, and Nathalie Japkowicz (2012). “The PerfSim algorithm for concept drift detection in imbalanced data”. In: 2012 IEEE 12th International Conference on Data Mining Workshops. IEEE, pp. 619–628. Archive, UCI KDD (2022). Information and Computer Science, University of California, Irvine, CA, USA. https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html. [Online; accessed 2022]. Baena-García, Manuel et al. (2006). “Early Drift Detection Method”. In: Proceedings of the 4th ECML PKDD International Workshop on Knowledge Discovery From Data Streams (IWKDDS’06). Barros, Roberto SM, Danilo RL Cabral, Paulo M Gonçalves Jr, and Silas GTC Santos (2017). “RDDM: reactive drift detection method”. In: Expert Systems with Applications 90, pp. 344–355. Barros, Roberto Souto Maior de, Juan Isidro González Hidalgo, and Danilo Rafael de Lima Cabral (2018). “Wilcoxon rank sum test drift detector”. In: Neurocomputing 275, pp. 1954–1963. Bartlett, Peter L, Shai Ben-David, and Sanjeev R Kulkarni (2000). “Learning changing concepts by exploiting the structure of change”. In: Machine Learning 41.2, pp. 153–174. Beck, Nathaniel, Jonathan N Katz, and Richard Tucker (1998). “Taking Time Seriously: Time-series-cross-section Analysis with a Binary Dependent Variable”. In: American Journal of Political Science 42.4, pp. 1260–1288. Benenson, Rodrigo, Markus Mathias, Tinne Tuytelaars, and Luc Van Gool (2013). “Seeking the strongest rigid detector”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3666–3673.Berkhahn, Felix et al. (2019). “Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un) supervised, and Supervised Learning”. In: arXiv preprint arXiv:1908.03015. Bifet, Albert and Ricard Gavalda (2007). “Learning from time-changing data with adaptive windowing”. In: Proceedings of the 2007 SIAM International Conference on Data Mining. SIAM, pp. 443–448. Bifet, Albert et al. (2013a). “Pitfalls in benchmarking data stream classification and how to avoid them”. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, pp. 465–479. Bifet, Albert et al. (2013b). “Pitfalls in Benchmarking Data Stream Classification and How to Avoid Them”. In: Proceedings of Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD. Ed. by Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný. Vol. 8188. Lecture Notes in Computer Science. Springer, pp. 465–479. Blythe, Duncan AJ, Paul Von Bunau, Frank C Meinecke, and Klaus-Robert Muller (2012). “Feature extraction for change-point detection using stationary subspace analysis”. In: IEEE Transactions on Neural Networks and Learning Systems 23.4, pp. 631–643. Boult, Terrance E et al. (2019). “Learning and the unknown: Surveying steps toward open world recognition”. In: Proceedings of the AAAI conference on artificial intelligence. Vol. 33. 01, pp. 9801–9807. Boyd, Kendrick, Kevin H Eng, and C David Page (2013). “Area under the precision-recall curve: point estimates and confidence intervals”. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp. 451–466. Brzezinski, Dariusz and Jerzy Stefanowski (2013). “Reacting to different types of concept drift: the accuracy updated ensemble algorithm”. In: IEEE Transactions on Neural Networks and Learning Systems 25.1, pp. 81–94.Brzezinski, Dariusz and Jerzy Stefanowski (2014). “Prequential AUC for classifier evaluation and drift detection in evolving data streams”. In: International Workshop on New Frontiers in Mining Complex Patterns. Springer, pp. 87–101.Bu, Li, Cesare Alippi, and Dongbin Zhao (2016). “A pdf-free change detection test based on density difference estimation”. In: IEEE Transactions on Neural Networks and Learning Systems.Bu, Li, Dongbin Zhao, and Cesare Alippi (2017). “An incremental change detection test based on density difference estimation”. In: IEEE Transactions on Systems, Man, and Cybernetics: Systems 47.10, pp. 2714–2726.Bucilua, Cristian, Rich Caruana, and Alexandru Niculescu-Mizil (2006). “Model compression”. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535–541.Bulatov, Yaroslav (2020). “NotMNIST dataset, 2011”. In: URL http://yaroslavvb. blogspot.com/2011/09/notmnist-dataset. html.Cai, Mu and Yixuan Li (2023). “Out-of-distribution detection via frequency-regularized generative models”. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5521–5530.Chen, Huanhuan, Peter Tiño, Ali Rodan, and Xin Yao (2014). “Learning in the Model Space for Cognitive Fault Diagnosis”. In: IEEE Trans. Neural Networks Learn. Syst. 25.1, pp. 124–136.Chen, Jiefeng et al. (2021). “Atom: Robustifying out-of-distribution detection using outlier mining”. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, pp. 430–445.Chen, Kylie, Yun Sing Koh, and Patricia Riddle (2016). “Proactive drift detection: predicting concept drifts in data streams using probabilistic networks”. In: IJCNN. IEEE, pp. 780–787.Chen, Wenhu, Yilin Shen, Hongxia Jin, and William Wang (2018). “A variational dirichlet framework for out-of-distribution detection”. In: arXiv preprint arXiv:1811.07308.150 Dasu, Tamraparni, Shankar Krishnan, Suresh Venkatasubramanian, and Ke Yi (2006).“An information-theoretic approach to detecting changes in multi-dimensional data streams”. In: In Proc. Symp. on the Interface of Statistics, Computing Science, and Applications. Citeseer.Dehghan, Mahdie, Hamid Beigy, and Poorya ZareMoodi (2016). “A novel concept drift detection method in data streams using ensemble classifiers”. In: Intelligent Data Analysis 20.6, pp. 1329–1350.Denouden, Taylor et al. (2018). “Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance”. In: arXiv preprint arXiv:1812.02765.DeVries, Terrance and Graham W Taylor (2018). “Learning Confidence for Out-of-Distribution Detection in Neural Networks”. In: arXiv preprint arXiv:1802.04865.Diers, Jan and Christian Pigorsch (2022). “Out-of-Distribution Detection Using Outlier Detection Methods”. In: International Conference on Image Analysis and Processing. Springer, pp. 15–26.Ditzler, Gregory and Robi Polikar (2011). “Hellinger distance based drift detection for nonstationary environments”. In: 2011 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE). IEEE, pp. 41–48.Dong, Fan, Jie Lu, Kan Li, and Guangquan Zhang (2017). “Concept drift region identification via competence-based discrepancy distribution estimation”. In: 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE). IEEE, pp. 1–7.Drummond, Nick and Rob Shearer (2006). “The open world assumption”. In: eSI Workshop: The Closed World of Databases meets the Open World of the Semantic Web. Vol. 15, p. 1.Duong, Quang-Huy, Heri Ramampiaro, and Kjetil Nørvåg (2018). “Applying temporal dependence to detect changes in streaming data”. In: Appl. Intell. 48.12, pp. 48054823.Elwell, Ryan and Robi Polikar (2011). “Incremental learning of concept drift in nonstationary environments”. In: IEEE Transactions on Neural Networks 22.10, pp. 15171531.Erdil, Ertunc, Krishna Chaitanya, and Ender Konukoglu (2020). “Unsupervised out-ofdistribution detection using kernel density estimation”. In: arXiv preprint arXiv:2006.10712. Faithfull, William J, Juan J Rodríguez, and Ludmila I Kuncheva (2019). “Combining univariate approaches for ensemble change detection in multivariate data”. In: Information Fusion 45, pp. 202–214.Fawcett, Tom (2006). “An introduction to ROC analysis”. In: Pattern recognition letters 27.8, pp. 861–874.Frias-Blanco, Isvani et al. (2014). “Online and non-parametric drift detection methods based on Hoeffding’s bounds”. In: IEEE Transactions on Knowledge and Data Engineering 27.3, pp. 810–823.Gal, Yarin and Zoubin Ghahramani (2016). “Dropout as a bayesian approximation: Representing model uncertainty in deep learning”. In: international conference on machine learning. PMLR, pp. 1050–1059.Gama, Joao, Pedro Medas, Gladys Castillo, and Pedro Rodrigues (2004). “Learning with drift detection”. In: Brazilian Symposium on Artificial Intelligence. Springer, pp. 286–295.Gama, João, Raquel Sebastião, and Pedro Pereira Rodrigues (2009). “Issues in evaluation of stream learning algorithms”. In: Proceedings of the 15th ACM SIGKDDO International Conference on Knowledge Discovery and Data Mining, pp. 329–338.Gama, João et al. (2014). “A survey on concept drift adaptation”. In: ACM computing surveys (CSUR) 46.4, p. 44.Gao, Jing, Wei Fan, Jiawei Han, and Philip S Yu (2007). “A general framework for mining concept-drifting data streams with skewed distributions”. In: Proceedings of the 2007 siam international conference on data mining. SIAM, pp. 3–14.Gao, Peichao, Zhilin Li, and Hong Zhang (2018). “Thermodynamics-based evaluation of various improved Shannon entropies for configurational information of gray-level images”. In: Entropy 20.1, p. 19.Geilke, Michael, Andreas Karwath, and Stefan Kramer (2015). “Modeling recurrent distributions in streams using possible worlds”. In: DSAA. IEEE, pp. 1–9.Gong, Dong et al. (2019). “Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1705–1714. Good, Phillip (2013). Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science & Business Media.Goodfellow, Ian et al. (2014). “Generative adversarial networks”. In: Communications of the ACM 63.11, pp. 139–144.Gu, Feng, Guangquan Zhang, Jie Lu, and Chin-Teng Lin (2016). “Concept drift detection based on equal density estimation”. In: 2016 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 24–30.Guo, Jia, Guannan Liu, Yuan Zuo, and Junjie Wu (2018). “An anomaly detection framework based on autoencoder and nearest neighbor”. In: 2018 15th International Conference on Service Systems and Service Management (ICSSSM). IEEE, pp. 16.Haque, Ahsanul et al. (2016). “Efficient handling of concept drift and concept evolution over stream data”. In: 2016 IEEE 32nd International Conference on Data Engineering (ICDE). IEEE, pp. 481–492.Härdle, Wolfgang and Léopold Simar (2007). Applied multivariate statistical analysis.Vol. 22007. Springer.Harel, Maayan, Shie Mannor, Ran El-Yaniv, and Koby Crammer (2014). “Concept drift detection through resampling”. In: International Conference on Machine Learning, pp. 1009–1017.Harries, Michael and New South Wales (1999). “Splice-2 comparative evaluation: Electricity pricing”. In: Technical Report.Hart, Peter E, David G Stork, and Richard O Duda (2000). Pattern classification. Wiley Hoboken.Heer, Matthäus et al. (2021). “The ood blind spot of unsupervised anomaly detection”.In: Medical Imaging with Deep Learning. PMLR, pp. 286–300.Hendrycks, Dan and Kevin Gimpel (2017). “A baseline for detecting misclassified and out-of-distribution examples in neural networks”. In: ICLR.Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich (2018). “Deep Anomaly Detection with Outlier Exposure”. In: International Conference on Learning Representations.Hilas, Constantinos S, Ioannis T Rekanos, and Paris Ast Mastorocostas (2013). “Change Point Detection in Time Series Using Higher-Order Statistics: A Heuristic Approach”. In: Mathematical Problems in Engineering 2013.Hoeffding, Wassily (1994). “Probability inequalities for sums of bounded random variables”. In: The collected works of Wassily Hoeffding. Springer, pp. 409–426. Hoens, T Ryan, Robi Polikar, and Nitesh V Chawla (2012). “Learning from streaming data with concept drift and imbalance: an overview”. In: Progress in Artificial Intelligence 1.1, pp. 89–101.Hsu, Yen-Chang, Yilin Shen, Hongxia Jin, and Zsolt Kira (2020). “Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951–10960.Huang, David Tse Jung, Yun Sing Koh, Gillian Dobbie, and Russel Pears (2014). “Detecting volatility shift in data streams”. In: 2014 IEEE International Conference on Data Mining (ICDM). IEEE, pp. 863–868.Huang, David Tse Jung, Yun Sing Koh, Gillian Dobbie, and Albert Bifet (2015). “Drift detection using stream volatility”. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, pp. 417–432.Huang, Yujia et al. (2019). “Out-of-distribution detection using neural rendering generative models”. In: arXiv preprint arXiv:1907.04572.Hulten, Geoff, Laurie Spencer, and Pedro Domingos (2001). “Mining time-changing data streams”. In: Proceedings of the seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 97–106. Hwang, Jenq-Neng, Shyh-Rong Lay, and Alan Lippman (1994). “Nonparametric multivariate density estimation: a comparative study”. In: IEEE Transactions on Signal Processing 42.10, pp. 2795–2810. Ioffe, Sergey and Christian Szegedy (2015). “Batch normalization: Accelerating deep network training by reducing internal covariate shift”. In: International Conference on Machine Learning. PMLR, pp. 448–456. Jain, Siddhartha, Ge Liu, Jonas Mueller, and David Gifford (2020). “Maximizing overall diversity for improved uncertainty estimates in deep ensembles”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. 04, pp. 4264–4271. Jiao, Yuchen, Yanxi Chen, and Yuantao Gu (2018). “Subspace change-point detection: A new model and solution”. In: IEEE Journal of Selected Topics in Signal Processing12.6, pp. 1224–1239. Katz-Samuels, Julian, Julia B Nakhleh, Robert Nowak, and Yixuan Li (2022). “Training ood detectors in their natural habitats”. In: International Conference on Machine Learning. PMLR, pp. 10848–10865. Kawahara, Yoshinobu, Takehisa Yairi, and Kazuo Machida (2007). “Change-point detection in time-series data based on subspace identification”. In: Seventh IEEE International Conference on Data Mining (ICDM 2007). IEEE, pp. 559–564. Khan, Shehroz S and Michael G Madden (2010). “A survey of recent trends in one class classification”. In: Artificial Intelligence and Cognitive Science: 20th Irish Conference, AICS 2009, Dublin, Ireland, August 19-21, 2009, Revised Selected Papers20. Springer, pp. 188–197.Kifer, Daniel, Shai Ben-David, and Johannes Gehrke (2004). “Detecting change in data streams”. In: VLDB. Vol. 4. Toronto, Canada, pp. 180–191.Kim, Youngin and Cheong Hee Park (2017). “An efficient concept drift detection method for streaming data under limited labeling”. In: IEICE Transactions on Information and Systems 100.10, pp. 2537–2546.Kingma, Diederik P and Max Welling (2013). “Auto-encoding variational bayes”. In: arXiv preprint arXiv:1312.6114.Kingma, Durk P and Prafulla Dhariwal (2018). “Glow: Generative flow with invertible 1x1 convolutions”. In: Advances in neural information processing systems 31. Kolter, J Zico and Marcus A Maloof (2007). “Dynamic weighted majority: An ensemble method for drifting concepts”. In: Journal of Machine Learning Research 8.Dec, pp. 2755–2790.Kotsiantis, Sotiris B (2013). “Decision trees: a recent overview”. In: Artificial Intelligence Review 39, pp. 261–283.Krizhevsky, Alex, Geoffrey Hinton, et al. (2009). “Learning multiple layers of features from tiny images”. In:Kubat, Miroslav and Kubat (2017). An introduction to machine learning. Vol. 2. Springer. Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell (2016). “Simple and scalable predictive uncertainty estimation using deep ensembles”. In: arXiv preprint arXiv:1612.01474.Langley, Pat (1996). Elements of machine learning. Morgan Kaufmann.Lazarescu, Mihai M, Svetha Venkatesh, and Hung H Bui (2004). “Using multiple windows to track concept drift”. In: Intelligent data analysis 8.1, pp. 29–59.LeCun, Y. (1998). The mnist database of handwritten digits.Lee, Kimin, Honglak Lee, Kibok Lee, and Jinwoo Shin (2017). “Training confidencecalibrated classifiers for detecting out-of-distribution samples”. In: arXiv preprint arXiv:1711.09325.Lee, Kimin, Kibok Lee, Honglak Lee, and Jinwoo Shin (2018). “A simple unified framework for detecting out-of-distribution samples and adversarial attacks”. In: Advances in neural information processing systems 31.Liang, Shiyu, Yixuan Li, and Rayadurgam Srikant (2017). “Enhancing the reliability of out-of-distribution image detection in neural networks”. In: arXiv preprint arXiv:1706.02690. Lin, Yih-Kai, Chu-Fu Wang, Ching-Yu Chang, and Hao-Lun Sun (2021). “An efficient framework for counting pedestrians crossing a line using low-cost devices: the benefits of distilling the knowledge in a neural network”. In: Multim. Tools Appl. 80.3, pp. 4037–4051.Liu, Ziwei, Ping Luo, Xiaogang Wang, and Xiaoou Tang (2015). “Deep learning face attributes in the wild”. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738.Losing, Viktor, Barbara Hammer, and Heiko Wersing (2016). “KNN classifier with self adjusting memory for heterogeneous concept drift”. In: 2016 IEEE 16th international conference on data mining (ICDM). IEEE, pp. 291–300.Lu, Jie et al. (2018). “Learning under concept drift: A review”. In: TKDE 31.12, pp. 23462363.Lu, Jie et al. (2020). “Learning under Concept Drift: A Review”. In: CoRR abs/2004.05785.Lu, Ning, Guangquan Zhang, and Jie Lu (2014). “Concept drift detection via competence models”. In: Artificial Intelligence 209, pp. 11–28.Lu, Ning, Jie Lu, Guangquan Zhang, and Ramon Lopez De Mantaras (2016). “A concept drift-tolerant case-base editing technique”. In: Artificial Intelligence 230, pp. 108133.Malinin, Andrey and Mark J. F. Gales (2018). “Predictive Uncertainty Estimation via Prior Networks”. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. Ed. by Samy Bengio et al., pp. 7047–7058.Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales (2019). “Ensemble distribution distillation”. In: arXiv preprint arXiv:1905.00076.Masana, Marc et al. (2018). “Metric learning for novelty and anomaly detection”. In:arXiv preprint arXiv:1808.05492.Mayo, Michael and Albert Bifet (2016). “Deferral Classification of Evolving Temporal Dependent Data Streams”. In: Proceedings of the 31st Annual ACM Symposium on Applied Computing, Pisa, Italy, April 4-8, 2016. Ed. by Sascha Ossowski. ACM, pp. 952–954. url: https://doi.org/10.1145/2851613.2851890.Mehrtens, Hendrik Alexander, Camila González, and Anirban Mukhopadhyay (2022).“Improving robustness and calibration in ensembles with diversity regularization”. In: arXiv preprint arXiv:2201.10908.Minku, Fernanda L and Xin Yao (2009). “Using diversity to handle concept drift in online learning”. In: 2009 International Joint Conference on Neural Networks. IEEE, pp. 2125–2132.Minku, Leandro L and Xin Yao (2012). “DDD: A new ensemble approach for dealing with concept drift”. In: IEEE Transactions on Knowledge and Data Engineering 24.4, pp. 619–633.Minku, Leandro L, Allan P White, and Xin Yao (2010). “The impact of diversity on online ensemble learning in the presence of concept drift”. In: IEEE TKDE 22.5, pp. 730–742.Morningstar, Warren et al. (2021). “Density of states estimation for out of distribution detection”. In: International Conference on Artificial Intelligence and Statistics. PMLR, pp. 3232–3240.Museba, Tinofirei, Fulufhelo Nelwamondo, Khmaies Ouahada, and Ayokunle Akinola (2021). “Recurrent adaptive classifier ensemble for handling recurring concept drifts”. In: Applied Computational Intelligence and Soft Computing 2021, pp. 1–13. Nair, Vinod and Geoffrey E Hinton (2010). “Rectified linear units improve restricted boltzmann machines”. In: International Conference on Machine Learning. Nalisnick, Eric et al. (2018). “Do deep generative models know what they don’t know?” In: arXiv preprint arXiv:1810.09136.Netzer, Yuval et al. (2011). “Reading digits in natural images with unsupervised feature learning”. In:NIPS Workshop on Deep Learning and Unsupervised Feature Learning. Nguyen, Anh, Jason Yosinski, and Jeff Clune (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images”. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427–436. Nguyen, Khanh Xuan and Brendan O’Connor (2015). “Posterior calibration and exploratory analysis for natural language processing models”. In: EMNLP. Nishida, Kyosuke and Koichiro Yamauchi (2007). “Detecting concept drift using statistical testing”. In: International conference on discovery science. Springer, pp. 264–269. Oord, Aaron Van den et al. (2016). “Conditional image generation with pixelcnn decoders”. In: Advances in neural information processing systems 29.Orabona, Francesco and Koby Crammer (2010). “New adaptive algorithms for online classification”. In: Advances in neural information processing systems 23.Oza, Nikunj C and Stuart J Russell (2001). “Online bagging and boosting”. In: International Workshop on Artificial Intelligence and Statistics. PMLR, pp. 229–236. Page, Ewan S (1954). “Continuous inspection schemes”. In: Biometrika 41.1/2, pp. 100115.Perera, Pramuditha and Vishal M Patel (2019). “Learning deep features for one-class classification”. In: IEEE Transactions on Image Processing 28.11, pp. 5450–5463. Pesaranghader, Ali and Herna L Viktor (2016). “Fast hoeffding drift detection method for evolving data streams”. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp. 96–111.Pesaranghader, Ali, Herna L Viktor, and Eric Paquet (2018). “McDiarmid drift detection methods for evolving data streams”. In: 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–9.Poor, H. Vincent (1996). Detection of abrupt changes: Theory and application : By Michèle and Igor V. Nikiforov. PTR Prentice-Hall, Englewood Cliffs, NJ (1993). ISBN 0-13-126780-9. Vol. 32. 8, pp. 1235–1236.Qahtan, Abdulhakim Ali, Basma Alharbi, Suojin Wang, and Xiangliang Zhang (2015). “A PCA-based change detection framework for multidimensional data streams: change detection in multidimensional data streams”. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Ed. by Longbing Cao et al. ACM, pp. 935–944. Quevedo, Joseba et al. (2014). “Combining learning in model space fault diagnosis with data validation/reconstruction: Application to the Barcelona water network”. In: Engineering Applications of Artificial Intelligence 30, pp. 18–29. Ramírez-Gallego, Sergio et al. (2017). “A survey on data preprocessing for data stream mining: Current status and future directions”. In: Neurocomputing 239, pp. 39–57. Rasmussen, Carl Edward and Joaquin Quinonero-Candela (2005). “Healing the relevance vector machine through augmentation”. In: Proceedings of the 22nd international conference on Machine learning, pp. 689–696. Reis, Denis Moreira dos, André Maletzke, Diego F Silva, and Gustavo EAPA Batista (2018). “Classifying and counting with recurrent contexts”. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1983–1992. Ren, Jie et al. (2019). “Likelihood Ratios for Out-of-Distribution Detection”. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Ed. by Hanna M. Wallach et al., pp. 14680–14691. Roberts, SW (1959). “Control chart tests based on geometric moving averages”. In: Technometrics 1.3, pp. 239–250. Ross, Gordon J, Niall M Adams, Dimitris K Tasoulis, and David J Hand (2012). “Exponentially weighted moving average charts for detecting concept drift”. In: Pattern recognition letters 33.2, pp. 191–198. Ruff, Lukas et al. (2018). “Deep one-class classification”. In: International conference on machine learning. PMLR, pp. 4393–4402. Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1985). Learning internal representations by error propagation. Tech. rep. California Univ San Diego La Jolla Inst for Cognitive Science.Saito, Kuniaki, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada (2018). “Maximum classifier discrepancy for unsupervised domain adaptation”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 37233732.Sakthithasan, Sripirakas, Russel Pears, Albert Bifet, and Bernhard Pfahringer (2015).“Use of ensembles of Fourier spectra in capturing recurrent concepts in data streams”. In: IJCNN. IEEE, pp. 1–8.Salganicoff, Marcos (1997). “Tolerating concept and sampling shift in lazy learning using prediction error context switching”. In: Lazy learning. Springer, pp. 133–155. Sarafijanovic-Djukic, Natasa and Jesse Davis (2019). “Fast distance-based anomaly detection in images using an inception-like autoencoder”. In: International Conference on Discovery Science. Springer, pp. 493–508.Schlimmer, Jeffrey C and Richard H Granger (1986). “Incremental learning from noisy data”. In: Machine learning 1.3, pp. 317–354.Sen, Pratap Chandra, Mahimarnab Hajra, and Mitadru Ghosh (2020). “Supervised classification algorithms in machine learning: A survey and review”. In: Emerging technology in modelling and graphics. Springer, pp. 99–111.Shafaei, Alireza, Mark Schmidt, and James J Little (2018). “A less biased evaluation of out-of-distribution sample detectors”. In: arXiv preprint arXiv:1809.04729. Shafaei, Alireza, Mark Schmidt, and James Little (2019). “A less biased evaluation of ood sample detectors”. In: Proceedings of the British Machine Vision Conference (BMVC).Shannon, Claude E (1948). “A mathematical theory of communication”. In: The Bell System Technical Journal 27.3, pp. 379–423.Shewhart, Walter Andrew (1931). Economic control of quality of manufactured product.Macmillan And Co Ltd, London.Sinha, Samarth et al. (2021). “Dibs: Diversity inducing information bottleneck in model ensembles”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 11, pp. 9666–9674.Sobolewski, Piotr and Michal Wozniak (2013). “Concept Drift Detection and Model Selection with Simulated Recurrence and Ensembles of Statistical Detectors.” In: J. UCS 19.4, pp. 462–483.Srivastava, Santosh et al. (2016). “Variance change point detection”. In: Proceedings of Indian Workshop on Machine Learning.Steiner, Stefan H (1999). “EWMA control charts with time-varying control limits and fast initial response”. In: Journal of Quality Technology 31.1, pp. 75–86.Sun, Linjin et al. (2021). “A new predictive method supporting streaming data with hybrid recurring concept drifts in process industry”. In: Comput. Ind. Eng. 161, p. 107625.Tanaka, Gouhei et al. (2019). “Recent advances in physical reservoir computing: A review”. In: Neural Networks 115, pp. 100–123.Tao, Qing, Dejun Chu, and Jue Wang (2008). “Recursive support vector machines for dimensionality reduction”. In: IEEE Transactions on Neural Networks 19.1, pp. 189193.Tsai, Du-Yih, Yongbum Lee, and Eri Matsuyama (2008). “Information entropy measure for evaluation of image quality”. In: Journal of digital imaging 21.3, pp. 338–347. Tuluptceva, Nina et al. (2020). “Anomaly detection with deep perceptual autoencoders”.In: arXiv preprint arXiv:2006.13265.Van Geert, Eline and Johan Wagemans (2017). “Objective and subjective complexityrelated measures and preferences for neatly organized compositions”. In: 40th European Conference on Visual Perception (ECVP), Berlin, Germany.Vapnik, Vladimir (1963). “Pattern recognition using generalized portrait method”. In:Automation and Remote Control 24, pp. 774–780.Vaze, Sagar, Kai Han, Andrea Vedaldi, and Andrew Zisserman (2021). “Open-set recognition: A good closed-set classifier is all you need”. In: arXiv preprint arXiv:2110.06207. Vernekar, Sachin et al. (2019). “Out-of-distribution detection in classifiers via generation”.In: arXiv preprint arXiv:1910.04241.Vyas, Apoorv et al. (2018). “Out-of-distribution detection using an ensemble of self supervised leave-out classifiers”. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 550–564.Wang, Haoliang, Chen Zhao, Xujiang Zhao, and Feng Chen (2022). “Layer Adaptive Deep Neural Networks for Out-of-Distribution Detection”. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, pp. 526–538.Wang, Heng and Zubin Abraham (2015). “Concept drift detection for streaming data”. In:2015 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–9. Wang, Senzhang, Jiannong Cao, and S Yu Philip (2020). “Deep learning for spatiotemporal data mining: A survey”. In: IEEE transactions on knowledge and data engineering 34.8, pp. 3681–3700.Wang, Shuo et al. (2013). “Concept drift detection for online class imbalance learning”.In: The 2013 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–10.Wang, Shuo, Leandro L Minku, and Xin Yao (2017). “A systematic study of online class imbalance learning with concept drift”. In: IEEE Transactions on Neural Networks and Learning Systems.Wang, Shuo, Leandro L. Minku, and Xin Yao (2018). “A Systematic Study of Online Class Imbalance Learning With Concept Drift”. In: IEEE Trans. Neural Networks Learn. Syst. 29.10, pp. 4802–4821.Wares, Scott, John Isaacs, and Eyad Elyan (2019). “Data Stream Mining: Methods and Challenges for Handling Concept Drift”. In: SN Applied Sciences 1.11, pp. 1–19. Webb, Geoffrey I et al. (2016). “Characterizing concept drift”. In: Data Mining and Knowledge Discovery 30.4, pp. 964–994.Wei, Hongxin et al. (2022). “Mitigating neural network overconfidence with logit normalization”. In: International Conference on Machine Learning.Wilcoxon, Frank (1992). “Individual comparisons by ranking methods”. In: Breakthroughs in statistics. Springer, pp. 196–202.Wu, Chen, Bo Du, and Liangpei Zhang (2013). “A subspace-based change detection method for hyperspectral images”. In: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 6.2, pp. 815–830.Xiao, Han, Kashif Rasul, and Roland Vollgraf (2017). “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms”. In: arXiv preprint arXiv:1708.07747. Xiao, Zhisheng, Qing Yan, and Yali Amit (2020). “Likelihood regret: An out-of-distribution detection score for variational auto-encoder”. In: Advances in neural information processing systems 33, pp. 20685–20696.Xu, Dan et al. (2015). “Learning deep representations of appearance and motion for anomalous event detection”. In: arXiv preprint arXiv:1510.01553.Xu, Shuliang and Junhong Wang (2017). “Dynamic extreme learning machine for data stream classification”. In: Neurocomputing 238, pp. 433–449.Yamanishi, Kenji and Jun-ichi Takeuchi (2002). “A unifying framework for detecting outliers and change points from non-stationary time series data”. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 676–681.Yang, Jingkang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu (2021). “Generalized out-ofDistribution detection: a survey”. In: arXiv preprint 2110.11334.Yeh, Arthur B, Richard N Mcgrath, Mark A Sembower, and Qi Shen (2008). “EWMA control charts for monitoring high-yield processes based on non-transformed observations”. In: International Journal of Production Research 46.20, pp. 5679–5699.Yu, Honghai and Stefan Winkler (2013). “Image complexity and spatial information”. In:2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).IEEE, pp. 12–17.Yu, Qing and Kiyoharu Aizawa (2019). “Unsupervised out-of-distribution detection by maximum classifier discrepancy”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9518–9526.Yu, Shujian, Xiaoyang Wang, and José C Príncipe (2018). “Request-and-reverify: hierarchical hypothesis testing for concept drift detection with expensive labels”. In:Proceedings of the 27th International Joint Conference on Artificial Intelligence. AAAI Press, pp. 3033–3039.Yu, Shujian et al. (2019). “Concept drift detection and adaptation with hierarchical hypothesis testing”. In: Journal of the Franklin Institute 356.5, pp. 3187–3215. Yuan, Yuan, Dong Wang, and Qi Wang (2016). “Anomaly detection in traffic scenes via spatial-aware motion reconstruction”. In: IEEE Transactions on Intelligent Transportation Systems 18.5, pp. 1198–1209.Zamprogno, Bartolomeu et al. (2020). “Principal component analysis with autocorrelated data”. In: Journal of Statistical Computation and Simulation 90.12, pp. 2117–2135. Zhang, Yuhong et al. (2017). “Three-layer concept drifting detection in text data streams”.In: Neurocomputing 260, pp. 393–403.Zhao, Yiru et al. (2017). “Spatio-temporal autoencoder for video anomaly detection”. In:Proceedings of the 25th ACM International Conference on Multimedia, pp. 19331941.Zhou, Chong and Randy C Paffenroth (2017). “Anomaly detection with robust deep autoencoders”. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665–674.Zliobaite, Indre (2013). “How good is the electricity benchmark for evaluating concept drift adaptation”. In: arXiv preprint arXiv:1301.3524.Zliobaite, Indre et al. (2015). “Evaluation methods and decision theory for classification of streaming data with temporal dependence”. In: Mach. Learn. 98.3, pp. 455–482. Zong, Bo et al. (2018). “Deep autoencoding gaussian mixture model for unsupervised anomaly detection”. In: International Conference on Learning Representations.