# ai-augmented-oral-assignment-design


## **Context and Rationale**

Traditional oral and "blue-book" assessments give faculty insight into students' authentic (AI-proof) understanding, but their in-person, labor-intensive nature makes them difficult to scale. Faculty cite time constraints, grading variability, and inequities in performance fluency as barriers. Without recording, there is no artifact to revisit—grading relies on notes and memory alone, making it nearly impossible to norm scores across evaluators or respond to grade disputes with evidence. We wanted to design a replicable, data-rich, AI-assisted workflow that preserves the integrity of oral exams while addressing these constraints.

This pilot extended the [Learning Lab's history of multimodal experimentation](https://hackmd.io/hv8Xy0qeTYCzpLpwJSezxQ?view). We selected a General Education course on folklore and performance as our test case. The course was a natural pedagogical fit, as students already engage in audio ethnography, oral storytelling, and analysis of performance.

The design integrates visual schemas (drawn cards and note-taking) with AI-generated follow-ups and automated data capture, producing an oral assessment that approaches students in multiple modes while maintaining scalablity.

## **Assignment Design**

Students entered a Learning Lab studio equipped with forward and overhead cameras (capturing both presentation and notes), a proctor for troubleshooting, and a card deck-based prompt system offering randomized conceptual combinations.

The workflow proceeded as follows:

- **Draw Cards:** Students drew four cards from each of three thematic decks—Core Concepts, Case Studies, and Theoretical Models.
- **Prepare (5 minutes):** Using a blank sheet and pencil, students selected up to five cards and planned a brief presentation. Preparation time and note-taking were recorded to ensure academic integrity.
- **Present (4–6 minutes):** Students delivered a spontaneous talk connecting their chosen cards.
- **AI-Generated Follow-Ups:** After each presentation, the system generated three follow-up questions derived from the student's transcript. Half were read aloud by a human; half by an AI voice (one of our research variables). These questions were not graded, but designed to probe comprehension and encourage reflection.
- **Survey:** Students completed a post-session survey on fairness, anxiety, and perceived learning, with over 90% response rate.

All session data—student name, drawn cards, transcript, and follow-up questions—were stored in a secure, human-readable database for analysis.

The assignment prompt provoded to students can be found [here](https://hackmd.io/7stnK8EbQKyYkKlKNU6otA?view).

## **Pedagogical Aims**

The oral format assessed conceptual/schematic understanding through unscripted synthesis, cognitive agility in linking theory, case, and concept under constraint, and performative literacy aligned with course themes on folklore, performance, and cultural transmission.

The AI component addressed historical pain points around norming and fairness (transcripts allow cross-grader calibration), scalability (automated scheduling, recording, and follow-up generation reduce human load), and reflective iteration (AI-generated questions create post-hoc learning artifacts for both students and instructors).

## **Pilot Implementation**

Two sample videos are below: a graduate fellow's (not in the course) demo illustrating the process and technical workflow, and the fauclty member (truncated due to scheduling constraints, not assignment limits).

*Note: student examples are withheld for FERPA, but with some pending permissions for research.*

<iframe width="560" height="315" src="https://www.youtube.com/embed/cyLTIwF_WzI?si=ws0mQrKNZXqcH0m4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

<iframe width="560" height="315" src="https://www.youtube.com/embed/vKcMtCtyyGk?si=HvGmzD61lsByutwJ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## **Preliminary Findings**

In one reflective survey (given one week after the exam), the data (n=74, >90% response rate) showed strong pedagogical outcomes but mixed student reception. 86% agreed exam preparation deepened understanding, 81% found topics accurately reflected content, and 73% judged difficulty appropriate. However, while 69% reported satisfaction, only 55% would choose this format again for the final, suggesting friction with implementation rather than the oral format itself (cited student concerns were centered around being recorded and the use of AI).

A second survey (immediately post-exam, n=75) focused specifically on the AI component and revealed similarly mixed results. 75% felt comfortable interacting with AI during Q&A, and 68% found AI questions helped them connect ideas. Students valued AI's role in deepening understanding (61% agreement), but 51% found the questions confusing or hard to understand, raising concerns about clarity and accessibility. This may be due to our test conditions, which focused on auditory reading of the questions by either one human and one AI voice (both held constant across all 80 exams). No text of the questions were displayed, which would be integrated for more clarity in future exams.

Qualitative feedback clarified student concerns: resistance centered on technology and environment, not the oral assessment itself. Students who praised the format simultaneously objected to elemts needed for scaling: recording, AI use, and the studio setting. As one wrote: "I liked the structure and content...However, I didn't like that it was recorded. I would much prefer to just sit down and talk to a person."

*Note: Full quantitative and process analysis is reserved for forthcoming research publication. Data sharing available upon request for collaborative research.*

## Example Questions:

Here is a random sampling of AI questions generated over the course of the exam period (nearly 300 were produced):

* When you said the hand-clap rhyme changed a little each time it was passed on, what’s one example of how you’ve heard it shift across groups?
* What’s one clear signal or “key” that lets participants and onlookers know the performance frame of the game has begun?
* When you described the Pigeon family’s work with the museum curator, what moment best captured their debate over what counts as an authentic basket?
* You mentioned the dark-water filter—what kind of rhetorical work do you think it’s doing to convey the sea’s unknown?
* Could you share one concrete sensory detail that women negotiate as part of the coffee aesthetics you described?
* You said class and gender shaped who felt comfortable in the café—can you give one moment that shows how those intersect?
* Could you share one practical technique an ethnographer could use to join a third space while still respecting the people there?
* Based on what you said about TikTok reshaping performance, how do you see a feature like *duets* changing who counts as a tradition bearer?
* What’s one local standard of beauty that makers use when judging whether a finished basket meets community expectations?
* Could you share one concrete way the twin laws of conservatism and dynamism appear in the modern teaching of Scottish smallpipes?
* You mentioned tourists asking for “more Mayan” patterns—what did that reveal to you about how the idea of authenticity gets negotiated?
* When you talked about string games as memory training, what specific move or moment shows that process most clearly?
* Could you share one everyday practice from your own routine that seems trivial but actually signals membership in a specific group?
* When you said hashtags helped Mamata Barbie reach new audiences, could you explain one concrete way that shifted how people saw her posts?
* Could you share one specific micro-political action that happens during the women’s coffee time you described?
* You mentioned a younger community member adding their own twist to a tradition—what did they change, and how did elders react?
* When you described harvesters asking permission from black ash trees, what detail in that ritual struck you most?
* Could you share one specific subreddit or online thread that, for you, captures the same social buzz as the old French salons?
* When you said the mortarboard decorations felt like a performance, what detail made you think so?
* When you mentioned memes speeding up variation online, how do you see that relating to the twin laws of tradition?