Investigating conversational agents to support secondary school computer science exploratory search

Abstract

Online educational resources (e.g., curricula, tutorials, documentation, Q\&A sites) increasingly serve as key sources for secondary school students learning Computer Science Principles (CSP). A big obstacle to using these resources is finding information appropriate for the learning task and learner's background. Research shows that secondary school students need support in searching the web and developing ``information literacy'' (e.g., find, evaluate, organize, use, and convey information). Exploratory search is particularly challenging as it goes beyond simple lookup and instead is comprised of learning and investigative search intents. ☐ A potentially promising approach uses a conversational search agent to conduct exploratory search through conversation with the learner. Research indicates that conversational agents can assist secondary school Computer Science (CS) teachers, especially those who lack CS background. In addition, research shows that secondary school students have high levels of engagement with conversational agents. Conversational agents could help alleviate social anxiety for students who do not want to ask teachers questions publicly. If designed with customization toward building better connections to learning, the agent could help students relate to the CS concepts to reflect on their own experiences as they relate to the CS concepts. ☐ The focus of this dissertation is Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search. The dissertation's contributions include: (1) the design of a Web-based generative conversational search agent tailored to the CS domain using SE word embeddings, (2) an exploratory study with 18 CS students investigates the agent's potential in supporting CS-related exploratory search, (3) a pedagogical fixed-response conversational search agent is designed for the CSP domain, aligning with Kaddoura's Think-Pair-Share collaborative learning strategy, (4) metrics for evaluating conversational agent effectiveness and engagement, (5) a comparative study involving 45 secondary school students in a CSP class explores the use of conversational agents and web search, (6) a knowledge-based pedagogical generative conversational search agent for the CSP domain, utilizing retrieval augmented generation and prompt engineering, (7) an exploratory study with 20 CSP students examining the customization's impact on aiding students in learning CSP concepts, and (8) an experience report of successes and challenges for future conversational agent researchers. ☐ The results of this dissertation show that CS students do find generative conversational agents useful at helping with exploratory search, yet CS students believe that the current generation of Web-based generative conversational search agents are not effective in helping with exploratory search. Evaluation of our approach using effectiveness and engagement metrics indicate that generative conversational agents are highly effective and interactive, and are preferred over pedagogical fixed-response conversational search agents, yet generative conversational agents pose risks as learning tools. Students are dissatisfied with the interaction and effectiveness of pedagogical fixed-response conversational search agents; however, find them helpful at finding useful information. Further, results indicate that students preferred a customized knowledge-based pedagogical generative conversational search agent, with its terminology more suitable to secondary school level, examples more understandable, and better connections to personal experiences compared to a standard generative conversational agent.