Behavioral Testing of deep neural knowledge tracing models

Recently, in the task of Knowledge Tracing(KT), Deep Neural Networks (DNN) showed superb performance over classical methods on multiple dataset benchmarks. While most Deep Learning based Knowledge Tracing (DLKT) models are optimized for general objective metrics such as accuracy or AUC on benchmark data, proper deployment of the service in a real-world environment requires additional qualities.

Definition:

Deep Learning based Knowledge Tracing (DLKT) uses Deep Neural Networks (DNN) to perform Knowledge Tracing(KT) which predicts how a student will respond to an unsolved question.

In this context, we adopt the idea of behavioral testing from software engineering and define desirable KT model behaviors. We propose an analysis framework to diagnose the KT model’s behavioral quality. This ensures that the model meets expectations and resigns from abnormal behaviors. Having test-run the framework on various datasets, the results highlight the impact of dataset size and model architecture upon the model’s behavioral quality.