Links: Paper | Poster | Talk

Venues:

• Proceedings of the 19th International Conference on Cognitive Modeling

• Abstract in proceedings of the 43rd Annual Meeting of the Cognitive Science Society, Vienna, Austria, 2021

Authors: Rebekah Gelpi, Nayan Saxena, George Lifchits, Daphna Buchsbaum & Christopher G Lucas

Abstract: People are capable of learning diverse functional relationships from data; nevertheless, they are most accurate when learning linear relationships, and deviate further from estimating the true relationship when presented with non-linear functions. We investigate whether, when given the opportunity to learn actively, people choose samples in an efficient fashion, and whether better sampling policies improve their ability to learn linear and non-linear functions. We find that, across multiple different function families, people make informative sampling choices consistent with a simple, low-effort policy that minimizes uncertainty at extreme values without requiring adaptation to evidence. While participants were most accurate at learning linear functions, those who more closely adhered to the simple sampling strategy also made better predictions across all non-linear functions. We discuss how the use of this heuristic might reflect rational allocation of limited cognitive resources.

Links: Paper | Code

Venue: Proceedings of the 21st IEEE International Conference on Advanced Learning Technologies, Tartu, Estonia, 2021

Authors: Nayan Saxena, Varun Lodaya & Trisha Thakur

Abstract: In this paper, we explore an extension of the Item Response Theory (IRT) model to predict student responses using dichotomous data and formulate approaches to improve the predictive accuracy of the traditional algorithm. We present a simple extension to the IRT modelling approach called IRT++, which combines both the 1-parameter and 2-parameter IRT models and modulates parameter optimisation through simple machine learning techniques like adaptive gradient descent and random-normal initialisation of parameters. By experimentation on real-world education data, we show how the IRT++ modelling framework other baselines at predicting student responses, and achieves better performance while sacrificing very little in model interpretability and rate of convergence.

Links: Paper | Preprint | Spotlight Talk | Code

Venue: Workshop on Reinforcement Learning for Education, 14th International Conference on Educational Data Mining , Paris, France, 2021 (Spotlight Paper)

Authors: Nayan Saxena, Pan Chen, Emmy Liu

Abstract: Multi-Armed-Bandit frameworks have often been used by researchers to assess educational interventions, however, recent work has shown that it is more beneficial for a student to provide qualitative feedback through preference elicitation between different alternatives, making a dueling bandits framework more appropriate. In this paper, we explore the statistical quality of data under this framework by comparing traditional uniform sampling to a dueling bandit algorithm and find that dueling bandit algorithms perform well at cumulative regret minimisation, but lead to inflated Type-I error rates and reduced power under certain circumstances. Through these results we provide insight into the challenges and opportunities in using dueling bandit algorithms to run adaptive experiments.

Links: Preprint | Workshop Paper | Poster | Code

Venue: Workshop on Adversarial Machine Learning, 38th International Conference on Machine Learning, 2021

Authors: Robert Wu*, Nayan Saxena* & Rohan Jain* (*equal contribution)

Abstract: Deep learning has proven to be a highly effective problem-solving tool for object detection and image segmentation across various domains such as healthcare and autonomous driving. At the heart of this performance lies neural architecture design which relies heavily on domain knowledge and prior experience on the researchers' behalf. More recently, this process of finding the most optimal architectures, given an initial search space of possible operations, was automated by Neural Architecture Search (NAS). In this paper, we evaluate the robustness of one such algorithm known as Efficient NAS (ENAS) against data agnostic poisoning attacks on the original search space with carefully designed ineffective operations. By evaluating algorithm performance on the CIFAR-10 dataset, we empirically demonstrate how our novel search space poisoning (SSP) approach and multiple-instance poisoning attacks exploit design flaws in the ENAS controller to result in inflated prediction error rates for child networks. Our results provide insights into the challenges to surmount in using NAS for more adversarially robust architecture search.