Lecture, Wed Mar 30

--- tags: nlp, 298, --- # Lecture, Wed Mar 30 The main aim is to discuss possible ideas for the final project. A description of the project is available in canvas (see Assignments). ## Project Ideas Let us extend the list of possible project ideas. - Use Grammatical Framework to build a recipe translator. - Use a language model for one of various NLP tasks such as sentiment analysis, question answering, topic detection, ... a great lists of [example projects at McCormick's blog](http://mccormickml.com/archive/), see also our [curated list](https://hackmd.io/@GAbY-V7cQxaoWKC3MkfoxQ/S1dW_z-mc). - What is the effect of fine tuning on question answering? [^question-answering] - Are there ways to make GPT-3 (or another language model) represent a person? [^person] - A version of Trivial Pursuit "Can you beat the AI?". For example: Use a scraper to extract Wikipedia pages on a topic. Finetune BERT with the Wikipedia data. Use Name Recognition to extract sentences with names. Mask out the names and present them on a screen. Make the human user fill in the blank and make BERT fill in the blank. Compare the result. Iterate. - Can we measure the distance between languages by using Google Translate (or OpenAI) to translate back and forth between two languages and then measure the similarity? [^fixedpoint] [^measure] - ... (let us know about your ideas) ... ## References and Further Reading - [McCormick's Notebooks](http://mccormickml.com/archive/) - Dive into Deep Learning (with notebooks) - [Chapter 14.8](https://d2l.ai/chapter_natural-language-processing-pretraining/bert.html) - [Chapter 15.6-7](https://d2l.ai/chapter_natural-language-processing-applications/finetuning-bert.html) ## <font color=red>Homework</font> Make preliminary plans for a project. See the Assignments in Canvas and sign up until **Fri April 8.**  [^question-answering]: For example, curate a list of questions on a certain topic, say the American Constitution. Collect the answers. Now fine-tune the language model with the Wikipedia pages on that topic and repeat the experiment. Do you get better answers in the 2nd case? See [here](https://colab.research.google.com/drive/1uSlWtJdZmLrI3FCNIlUHFxwAJiSu2J0-#scrollTo=6_mAnIPKaXyw) and [here](https://colab.research.google.com/drive/1k7lExKPaZyI9LKeeQFpZeF92hFNCQ5nC#scrollTo=r-byttpdMhJw) and [here](https://towardsdatascience.com/question-and-answering-with-bert-6ef89a78dac) and [here](https://developer.apple.com/documentation/coreml/model_integration_samples/finding_answers_to_questions_in_a_text_document) for using BERT (a limitation is that text and question are limited to a few hundred tokens). [^fixedpoint]: It is also interesting to measure how long it takes to reach fixed point. [^measure]:(Remember that measuring similarlity can be done with language models.) [^person]: It is easy to trick GPT-3 into giving contradictory answers about age, gender, nationality, etc. What happens if one fine-tunes the language model with question-answer pairs? Does the fine-tuned language model only learn the answers to the specific quesitons used during the fine-tuning, or does it start building a coherent persona?