| Course Subject: | Data studies and game-related courses |
| Student Level: | Graduate students |
| Number of Students: | NA |
| Developed by: | Adriana de Souza e Silva, Professor, Communication Studies, College of Arts, Media, and Design; Director of the Center for Transformative Media at the College of Arts |
What the Instructor Did
The instructor designed, built, and piloted a generative AI chatbot (named Pixel) that was embedded in an existing web archive of mobile games and devices. Pixel was built to serve as a guided educational resource that helps users interact with the site and get answers to their questions in natural language.
Purpose
The archive contains rich material on the history of mobile games and devices, but its structure requires users to already know something about the collection in order to navigate it productively. Pixel was designed to lower that barrier. It can help students find relevant devices, understand historical context, and even contribute to the collection. The interactive nature of the tool can support student learning, motivation, and curiosity about games.
Assessment
System administrators will be able to access anonymized interaction logs to identify points of confusion along with other usage analytics and metrics.
Faculty Reflections
A recent pilot testing session was conducted with four graduate students who tested the tool. The session confirmed that Pixel works well as an entry point into a complex archive, particularly for users who are unfamiliar with the collection’s scope and structure. Students found the bot’s personality distinct from general-purpose AI tools and noted that it generated consistent results. That consistency is important for use as a teaching tool, as it creates the necessary trust that the bot’s answers faithfully reflect the archive’s contents.
Step-by-Step Directions
| Step 1 | Define the chatbot’s core functions and educational goals before beginning development. For the RMGP chatbot (“Pixel”), this meant specifying four target behaviors: searching and returning relevant games from the archive database, providing accurate collection and mission information, responding in under three seconds, and functioning on both desktop and mobile. Document these criteria explicitly so they can serve as the basis for usability testing. |
| Step 2 | Build the chatbot with access to the archive database and the ability to navigate site structure through natural conversation. Design a persona that helps non-experts build a mental map of the collection — for RMGP, this took the form of a light “game-world guide” named Pixel that frames navigation through eras (e.g., “early Nokia era,” “J2ME corridor”) and uses storytelling to orient users unfamiliar with mobile gaming history. Program graceful handling for out-of-scope queries: the bot should stay in character and redirect users toward what the archive does contain, rather than returning a bare error. |
| Step 3 | Develop a structured usability testing protocol before the testing session. Specify the query types testers should try — for example, in-scope titles, out-of-scope titles, historical questions, and contribution-related questions. Prepare a simple feedback framework covering response accuracy, response speed, tone and personality, and handling of edge cases. Having explicit criteria in advance makes it possible to compare results across testers and sessions, and to distinguish technical problems from design problems. |
| Step 4 | Recruit a small pilot group whose background is relevant to both the tool and its intended use. For the RMGP pilot, four graduate students enrolled in MSCR 6310 (Critical Data Studies) served as testers. Their familiarity with digital archives and critical data practices meant they could identify not only usability problems but also deeper issues around scope, representation, and what the collection implicitly includes or excludes. |
| Step 5 | Run a testing session. Ask participants to interact with the chatbot using both their own questions and a set of prepared prompts covering a range of query types. Observe response times, retrieval accuracy, and how the bot handles queries outside the archive’s scope. In the RMGP pilot session on March 30, 2026, Pixel handled 30+ queries and met the under-three-second response target. In-scope titles such as Metroid: Zero Mission (2004) were retrieved successfully; out-of-scope titles such as Assassin’s Creed: Black Flag and Dragon Ball Z returned appropriate null results. |
| Step 6 | Collect structured feedback from testers immediately after the session. Ask about response accuracy, tone and personality, consistency across varied phrasings, and overall usability. In the RMGP pilot, students noted that Pixel felt distinct from general-purpose AI tools. One student observed that encountering the bot’s limits naturally surfaced the logic of the collection itself: why certain titles are absent, and what counts as a “mobile game” by the archive’s definition. This was unplanned but turned out to be one of the most pedagogically valuable outcomes of the testing session. |
| Step 7 | Review results against your pre-specified criteria and identify concrete areas for iteration. For RMGP, the key findings were: (1) students’ existing game knowledge frequently led them outside the 1975-2008 window, so the activity design needs to anticipate and frame those moments productively rather than treat them as dead ends; (2) archive metadata should be expanded to improve similarity-based search; and (3) the eventual classroom assignment needs clearer guidance for students when the bot cannot find what they are looking for. Document planned next steps before designing the student-facing learning activity. |
| Step 8 | Use what you learned in the testing session to design the student learning activity. Draw directly on what testing revealed about how users naturally interact with the tool — what they ask first, where they get stuck, and what surprises them. For RMGP, the planned learning activity will ask students to investigate devices and games from different eras using Pixel as a guide, then repeat the same task without the chatbot, and reflect on how the two experiences compared. |
Related Materials
Download PDF version of this assignment
Return to AI Gallery Home