---
title: 'Primer on GPT Family of models'
disqus: hackmd
---

{Chat/Instruct}GPT - Behind the scenes
===

## Table of Contents

[TOC]

## What is GPT?

* Transformer model (only decoder) that does language modeling.
* Basically, predicts the nth word given (n-1) words as context.
* 
* While language modeling has been around since 2000s, GPT showed that by training a model on the whole internet (Common Crawl), we unlock remarkable capabilities.

### GPT-3


#### Problems with this approach?

* Toxicity
* Cannot understand human notions of which answer is appropriate or not?
* Very sensitive to structure of prompt

### Aligning GPT with user expectations - Instruct GPT


* ChatGPT is essentially InstructGPT, but trained with extremely long context (8192 tokens).

## The unexpected :shocked_face_with_exploding_head:

* Can simulate a linux terminal/vm

* Inception - create a VM from within the VM



## The Good

* Amazing at solving Leetcode problems/Pair programming


* Can explain complex stuff quite well


* Can reason about a wide range of stuff, even economics


* Can even create text based games.


## The Bad

* Struggles at basic math


* Generalization still not grounded.


* Eager to please the user


* Very funny exchange on "what is the fastest marine mammal?"


* Lots of work to be done in grounding


## The ugly

* People find a way to bypass most safety filters


* Spews text that looks plausible, but need not be correct. Makes it 100x difficult for moderators of crowd-sourced platforms.


### Pitfalls associated wth Instruction Finetuning

#### Amplification of Bias

##### 1. Lack of diversity in Answers
* GPT


* Instruct GPT


#### 2. Some form of mode collapse happens

* GPT


* InstructGPT


#### 3. Optimization difficulties

* They accidentally overoptimized a GPT policy against a positive sentiment reward model. This policy evidently learned that wedding parties were the most positive thing that words can describe, because whatever prompt it was given, the completion would inevitably end up describing a wedding party.

In general, the transition into a wedding party was reasonable and semantically meaningful, although there was at least one observed instance where instead of transitioning continuously, the model ended the current story by generating a section break and began an unrelated story about a wedding party.

## Model Editing

(https://arxiv.org/abs/2110.11309)