# hum-350-visit-20251107
## tinyurl.com/hum-350


## Links

(If you ever want to do a deeper dive on this, we'd be happy to run another workshop, or you can check out the [slides that we show all of your TFs during our August Training](https://hackmd.io/SfZ8F08oQGaodOkThCa0Yw?view)). You should also take a look at the Bok Center's resources on [Teaching in the Age of AI](https://bokcenter.harvard.edu/artificial-intelligence), which is where we are sending the faculty whose courses you frequently teach.

## Agenda
* Intros
* Defense against the Dark Arts
* Using AI for Our Work (Joining Slytherin?)

## Intro

What are we doing with AI right now? And how do we feel about it?

## Defense Against the Dark Arts (A Case Study)

### GENED 1196's Original Paper Assignment

Like a lot of Humanities courses, GenEd 1196 relied on a [midterm essay](https://docs.google.com/document/d/1jy5XH6So-y75Mp4OX5_oo1F0Gm_poytW/edit?usp=drive_link&ouid=101521386334738626081&rtpof=true&sd=true) in the past as a core feature of the course.

### Red-Teaming

Assignments of this sort can be vulnerable to AI-misuse, and one way of test this is by "Red-Teaming," where we use AI to try and perform tasks involved in a given assignment, in order to gauge just how resilient it is.

#### The Most Basic Prompting

I the most basic case, a student might just put in the essay prompt and ask ChatGPT or Gemini to [“write my paper on X”](https://hackmd.io/KgQuhSvmTlGdz1nnXdJsEw?view)

#### Can We Detect It?

Tools such as [GPTZero](https://app.gptzero.me/) can sometimes do a good job of detecting unedited ChatGPT output (and can also give you a list of some of the [most common ChatGPT phrases!](https://gptzero.me/ai-vocabulary)).

But research has demonstrated that they can generate both false positives and false negatives, and, as we can see by playing around with it, it is relatively easy to take something that looks to be 100% AI-generated and, by tweaking just a few lines, have it evaluated as 100% human.

#### Advanced Prompting and "Context Engineering"

Now there are much more advanced ways to use AI to write an essay, and if we are really going to Red Team this assignment, we are going to want to go all out.

But in order to understand the what and the why of the next steps, it helps to just remind ourselves of a few of the pecularities of how these models "think."



So what we need to do is produce MORE and BETTER context for the LLM. Because without that context, it is essentially like a student who hasn't done the reading. We'll get responses that sound reasonable, but they'll be untethered from the actual texts and ideas we want to work with.

What does this look like?



Well, our Red Team might use use **multiple** interactions with ChatGPT and other AI tools to generate the context that will help yield a better outcome. For instance...

* [some relevant quotes from secondary source materials](https://hackmd.io/MqJNf0H5T36e6pwZyFkecw?view)
* [a personal narrative](https://hackmd.io/979SjjhzQBye-mGOn8XgcQ?view)
* [a first draft (with savvy prompting)](https://hackmd.io/4boHLNiVQLq5BFsO4e3Jxw?view)
* [a critical "negation" of that first draft](https://hackmd.io/MfacOTmlQASAB8U2usIr-Q?view)
* [and then a negation of negation](https://hackmd.io/5XFLaVxNTpKMvQh7SK9E5w?view), by which point we'll have a much better quality paper (and you can imagine this process extending in multiple directions with multiple recursive steps).


---

## Try it out

Try your hand at redteaming the assignment prompts. Feel free to use the above materials or source your own, and select a paper prompt from the [essay assignment](https://docs.google.com/document/d/1jy5XH6So-y75Mp4OX5_oo1F0Gm_poytW/edit?usp=drive_link&ouid=101521386334738626081&rtpof=true&sd=true) or the even the short essays from [the exam](https://docs.google.com/document/d/1J9_o4BMMbaOJmX1gS0r2Ybf7uL5NOnlA/edit#bookmark=id.wa9v95iu1dnx).

---

### The Assignment (Re)Design

The Bok Center has been working to develop guidance for faculty on coping with teaching in the Age of AI (see [this link](https://bokcenter.harvard.edu/courses-and-assignments-in-age-of-ai) for a whole bunch of ideas), and one of the ideas that really resonated with this instructor was an oral exam (for more examples of oral exams we've supported in our Learning Lab, check out [this link](https://hackmd.io/hv8Xy0qeTYCzpLpwJSezxQ?view)).

But while traditional oral exams let instructors gauge real understanding, but they don’t scale. Faculty report as challenges:
- **Time pressure** and **grading inconsistency**
- **Performance bias** favoring fluent speakers
- **No artifact** for calibration or review

So we decided, in this pilot, to include
- a video recording of the student taking the exam,
- an AI-generated transcription of the exam,
- and AI-generated follow-up questions based on the students' initial answers


The full writeup of the design is [here](https://hackmd.io/MnP3Vc0oSNiH5EEi1O1wYQ?view)

---

## Using AI for Our Own Work?

Are there any aspects of the "Red Teaming" process that are legitimate moves for academic researchers? (or teachers?)

Let's think about some ways we could use leveled-up gen-AI-skills for teaching and research

### Context Context Context

* discussion of context management
* current strategies they use
* ethical uses of AI to help with this that they could imagine, including comfort levels with
* AI summaries
* finding sources
* lists of archives
* etc.
* animating question here: what prompting systems work for your materials?

### Structured Outputs & Even Code

In OpenAI's [Agent Builder](https://platform.openai.com/agent-builder).

## Coding Stations

### Analyzing with code

Structure outputs can be used in various textual operations. We have two examples:
* [Concordances](https://colab.research.google.com/drive/1n_26uMBc-4H9Ij_dYZCHaiUbWabt6-EF?usp=drive_link)
* [Poetry -> structured output -> generated poetry -> audio](https://colab.research.google.com/drive/15INMpqpKFMGIkr4SB8vWs1-WxBKD18ES?usp=drive_link)

To explore this yourself we have two python notebooks:
* [Pushing Poetry to a Database](https://colab.research.google.com/drive/1eGUHn-IyyeR8LKX1VoeKcoGNiqqJybiN#scrollTo=X3w9_JhusGYx)
* [Pulling Poetry from a Database](https://colab.research.google.com/drive/1vzxiG_06IfxwWUYeNUMJ5chaeTPnQsGd#scrollTo=bde49cf0)

All of this data goes to [this airtable base](https://airtable.com/appoGrUBCmygROEX2/shrFi17qpc50LlhHl).

Feel free to make your own copy of these notebooks, and ask Gemini (embedded in the colabs) to explain the code to you. You can even try to cocreate the sort of prompt that would help you generate this code yourself.

### Displaying with code

If you want to display your research findings or particular concepts you're teaching in section, you could vibe code quick web apps that combine images, videos, text, and any other assets that you could imagine being useful.

For example, below is a vibe-coded prototype of the [Feral Atlas](https://feralatlas.org/).


Others:
- [a sample link to a visual essay](https://gened-1145.vercel.app/video-essay/02) from a cinema course workshop
- [an example of an interactive essay about video games]()

---

### Stay connected!

Feel free to reach out at [e-mail] for any AI-related inquiries.

A calendar of events and workshops can be found [here](https://bokcenter.harvard.edu/generative-ai-events).