Lecture, Wed Mar 30

The main aim is to discuss possible ideas for the final project.

A description of the project is available in canvas (see Assignments).

Project Ideas

Let us extend the list of possible project ideas.

• Use Grammatical Framework to build a recipe translator.

• Use a language model for one of various NLP tasks such as sentiment analysis, question answering, topic detection, … a great lists of example projects at McCormick's blog, see also our curated list.

  • What is the effect of fine tuning on question answering? ^[1]
  • Are there ways to make GPT-3 (or another language model) represent a person? ^[2]
  • A version of Trivial Pursuit "Can you beat the AI?". For example: Use a scraper to extract Wikipedia pages on a topic. Finetune BERT with the Wikipedia data. Use Name Recognition to extract sentences with names. Mask out the names and present them on a screen. Make the human user fill in the blank and make BERT fill in the blank. Compare the result. Iterate.

• Can we measure the distance between languages by using Google Translate (or OpenAI) to translate back and forth between two languages and then measure the similarity? ^{[3] [4]}

• … (let us know about your ideas) …

References and Further Reading

• McCormick's Notebooks
• Dive into Deep Learning (with notebooks)
  • Chapter 14.8
  • Chapter 15.6-7

Homework

Make preliminary plans for a project. See the Assignments in Canvas and sign up until Fri April 8.

---

1. For example, curate a list of questions on a certain topic, say the American Constitution. Collect the answers. Now fine-tune the language model with the Wikipedia pages on that topic and repeat the experiment. Do you get better answers in the 2nd case? See here and here and here and here for using BERT (a limitation is that text and question are limited to a few hundred tokens). ↩︎

2. It is easy to trick GPT-3 into giving contradictory answers about age, gender, nationality, etc. What happens if one fine-tunes the language model with question-answer pairs? Does the fine-tuned language model only learn the answers to the specific quesitons used during the fine-tuning, or does it start building a coherent persona? ↩︎

3. It is also interesting to measure how long it takes to reach fixed point. ↩︎

4. (Remember that measuring similarlity can be done with language models.) ↩︎