Addressing Selection Bias in Computerized Adaptive Testing: A User-Wise Aggregate Influence Function Approach

Computerized Adaptive Testing (CAT), the core technology behind R.test, adaptively selects questions that align with the test taker’s proficiency, minimizing redundancy and irrelevant questions while ensuring precise measurement. CAT relies on pre-trained item profiles, developed from extensive collections of dense item-response data. The quality of these initial profiles significantly impacts CAT’s performance.

Our paper explores the use of CAT response data collected from our service to continuously improve CAT performance. We introduce a novel debiasing method known as the User-wise Aggregate Influence Function approach (UserAIF). This method effectively mitigates the selection bias present in CAT response data, and overcomes the limitations of traditional approaches. Our experiments reveal that incorporating CAT response data through UserAIF enhances CAT performance, even with limited unbiased initial data.