---
# System prepended metadata

title: RTiAI final presentation

---

# RTiAI final presentation

- Total length: 13 min + 4 min Q&A

## Motivation, Task and Research Question

### Meta

- Time: 1 min
- Presenter: Benjamin

### Content

- Keep old slide

## Datasets

### Meta

- Time: 1min
- Presenter: Benjamin

### Content

- Pushshift API
    - Pros: Publicly Available, Loose Rate Limiting
    - Cons: Does not contain up-to-date data, rather one time snapshots
- Reddit API
    - Pros: Contains up-to-date data
    - Cons: API Key required, hard rate limiting
- Solution: Combination of both

## Overview

### Meta

- Time: 3 min
- Presenter: Nikola

### Content

- Some concrete examples
- GPT2 and GPTNeo
    - Average loss table
- Sentence Complexity table
- Clustering
- Survey

## Answers to Research Questions

### Meta

- Time: 3 min
- Presenter: Nikola

### Content

- Scientific Analaysis of Results

## Interesting Challenges

### Meta

- Time: 2 min
- Presenter: Benjamin

### Content

- Large Language Models
    - Problem: Does not fit onto one GPU
    - Approaches: Tried reducing precision, split up to multiple GPUs
    - Finally: Didn't get it working, used a smaller model
    - NANs in training
        - Different optimizer (Adafactor), manually tweaking the LR
- Pushshift API
    - Problem 1: Very unstable / not available at all
    - Solution 1: Integrate refetching mechanism
    - Problem 2: Pushshift made a huge update that made data before November 2 inaccessible
    - Approaches 2: Considered Reddit API only (would have taken over two weeks two fetch with their rate limiting), ParseHub, found Pushshift Data Dump (multiple TB of data)
    - Solution 2: Fetch Data Dump one item at a time, filter out relevant data, delete after each iteration (no more than 200GB at a time)

## Future Work within the Seminar

### Meta

- Time: 1 min
- Presenter: Benjamin

### Content

- Final evaluation of Showerthoughts Survey
- Finish ELI5 Model
- Survey for ELI5
- Better Topic Analysis (for Clusters)
- Write Paper

## Potential Future Work

### Meta

- Time: 1 min
- Presenter: Benjamin

### Content

- Are our generated Showerthoughts bot detectable? Can this models maybe even be used to improve Bot Detection? Maybe a GAN architecture could be used for this?
- Is it possible to create a language model that can generate posts for any subreddit (train on entire reddit data)? How will it perform compared to a specific fine-tuned one?
- Question: At what point do Language Models not even need fine-tuning for specific tasks as generating Showerthoughts? ChatGPT is a fairly large model and yet didn't excel at this task. Will GPT-4, for example, be able to do this without fine-tuning?
- Measuring how funny something is. There is fairly good approaches to sentiment, etc. but "funny" seems to be very tricky.
    - Whether or not something is funny is very individual. We all got a different sense of humor. However, in specific subreddits, mutual people come along and there are posts that get upvoted/downvoted a lot. So there seems to be a pattern. Can this be replicated by AI?
