# Defense Against the Dark Arts: AI for Skeptics

## tinyurl.com/mahindra-dark-arts

### Thursday, November 6, 2025 · 12:00pm

Beginning with the assumption that AI skeptics need to understand how it works in order to **defend against it**, this session walks through a case study of a course the Bok Center partnered with this term to help us achieve this goals:

* how AI tools can respond to current assignments
* some of the ways we might guard against this to preserve core humanities skills of **reading, writing, and discussion**.
* discuss next steps for partnering with you in the defense against the Dark Arts
---

## Case Study: GENED 1196

### The Classic Assignment

Like a lot of Humanities courses, GenEd 1196 relied on a [midterm essay](https://docs.google.com/document/d/1jy5XH6So-y75Mp4OX5_oo1F0Gm_poytW/edit?usp=drive_link&ouid=101521386334738626081&rtpof=true&sd=true) in the past as a core feature of the course.

### Red-Teaming

Assignments of this sort can be vulnerable to AI-misuse, and one way of test this is by "Red-Teaming," where we use AI to try and perform tasks involved in a given assignment, in order to gauge just how resilient it is.

#### The Most Basic Prompting

In the most basic case, a student might just put in the essay prompt and ask ChatGPT or Gemini to [“write my paper on X”](https://hackmd.io/KgQuhSvmTlGdz1nnXdJsEw?view)

#### Can We Detect It?

Tools such as [GPTZero](https://app.gptzero.me/) can sometimes do a good job of detecting unedited ChatGPT output (and can also give you a list of some of the [most common ChatGPT phrases!](https://gptzero.me/ai-vocabulary)).

But research has demonstrated that they can generate both false positives and false negatives, and, as we can see by playing around with it, it is relatively easy to take something that looks to be 100% AI-generated and, by tweaking just a few lines, have it evaluated as 100% human.

#### Advanced Prompting and "Context Engineering"

Now there are much more advanced ways to use AI to write an essay, and if we are really going to Red Team this assignment, we are going to want to go all out.

But in order to understand the what and the why of the next steps, it helps to just remind ourselves of a few of the pecularities of how these models "think."

(If you ever want to do a deep dive on this, we'd be happy to run another workshop, or you can check out the [slides that we show all of your TFs during our August Training](https://hackmd.io/SfZ8F08oQGaodOkThCa0Yw?view))



So what we need to do is produce MORE and BETTER context for the LLM. Because without that context, it is essentially like a student who hasn't done the reading. We'll get responses that sound reasonable, but they'll be untethered from the actual texts and ideas we want to work with.

What does this look like?



Well, our Red Team might use use **multiple** interactions with ChatGPT and other AI tools to generate the context that will help yield a better outcome. For instance...

* [some relevant quotes from secondary source materials](https://hackmd.io/MqJNf0H5T36e6pwZyFkecw?view)
* [a personal narrative](https://hackmd.io/979SjjhzQBye-mGOn8XgcQ?view)
* [a first draft (with savvy prompting)](https://hackmd.io/4boHLNiVQLq5BFsO4e3Jxw?view)
* [a critical "negation" of that first draft](https://hackmd.io/MfacOTmlQASAB8U2usIr-Q?view)
* [and then a negation of negation](https://hackmd.io/5XFLaVxNTpKMvQh7SK9E5w?view), by which point we'll have a much better quality paper (and you can imagine this process extending in multiple directions with multiple recursive steps).


so ....

### The Assignment (Re)Design

The Bok Center has been working to develop guidance for faculty on coping with teaching in the Age of AI (see [this link](https://bokcenter.harvard.edu/courses-and-assignments-in-age-of-ai) for a whole bunch of ideas), and one of the ideas that really resonated with this instructor was an oral exam (for more examples of oral exams we've supported in our Learning Lab, check out [this link](https://hackmd.io/hv8Xy0qeTYCzpLpwJSezxQ?view)).

But while traditional oral exams let instructors gauge real understanding, but they don’t scale. Faculty report as challenges:
- **Time pressure** and **grading inconsistency**
- **Performance bias** favoring fluent speakers
- **No artifact** for calibration or review

So we decided, in this pilot, to include
- a video recording of the student taking the exam,
- an AI-generated transcription of the exam,
- and AI-generated follow-up questions based on the students' initial answers


The full writeup of the design is [here](https://hackmd.io/MnP3Vc0oSNiH5EEi1O1wYQ?view)

---

## Some Design Patterns You Can Reuse Today

Our [Bok Center guidance](https://bokcenter.harvard.edu/courses-and-assignments-in-age-of-ai) offers some basic tenents of how to make assignments more resilient-- or you can cross to the "Dark Arts" and even use it in your course design.

### 1) AI-Resilient (low-or-no AI permitted)

These adjustments reduce the chance that AI can produce fluent but unreliable submissions, while staying aligned with the assignment’s learning goals:

#### Connect to course-specific materials
* Require students to explicitly reference lecture content, not just readings.
* Ask students to incorporate "local data" produced in the course or not otherwise available in the LLM training (i.e. discussion notes, a guest speaker’s remarks, recent case studies, objects/subjects/texts from Harvard collections unavailable to the public.)
* Have students provide page-specific citations or quotations, which prevents AI-generated generic summaries.
#### Add in-person or no-device checkpoints
* Require an in-class outline or oral proposal where students present their thesis and sources before drafting.
* After submission, include a short oral defense or reflective writing exercise (10–15 minutes, in section) where students explain their argument without the assistance of notes.
#### Add a process reflection piece
* Ask for a written memo or in-person checkin where they describe: how they developed their thesis, what counterarguments they considered, and which readings were most useful. The same can be done for more complex code.

### 2) AI-Enhanced (transparent AI use)

If your course allows AI, you can adapt the assignment to use it as a learning tool rather than a shortcut:
#### Transparent AI workflow log
* Require students to submit a short AI log (screenshots or structured handout).
* Logs can note what was helpful, what students rejected, and why.
#### Fact-checking task
* Students may generate AI-produced draft paragraphs or code but must revise and fact-check them using course sources.
* Their submission can include both versions (AI and revised) plus a rationale for the revision and the methods + results of their fact-checking steps.
#### Pair AI prep with no-AI verification
* Let students use AI for early generation, but require complementary short in-class written exercise (e.g., summarize their thesis or annotate/edit code in 15 minutes, no devices).

---

### Stay connected!

Feel free to reach out at [e-mail] for any AI-related inquiries.

A calendar of events and workshops can be found [here](https://bokcenter.harvard.edu/generative-ai-events).