---
# System prepended metadata

title: LLM-Code-a-thon2024

---


## Summary Notes 

### Wednesday Jan 10th, 2024

9:00 AM Nirav's introduction.

Mithun discussion: What are LLM? 
- what is intelligence?
- ChatGPT is not intelligent
- What are LLM?
- What is a Turing test?
- Can chatGPT pass a test?
- Why are we here today?
- What are the issues we will come up against?
- Privacy concerns: all information passed to ChatGPT is public, not good for HIPAA, FERPA and HSR proteted data
- "No guardrails" in terms of biases

10:00 AM

- [Ollama](https://github.com/jmorganca/ollama) Nirav demo running in CyVerse (running on [GPU 06:](http://gpu06.cyverse.org:8080))
- Goal for the codeathon: create an open source chat bot that resembles ChatGPT.
    - Allows:
        - Professors to upload data
        - Students to ask questions
    - Timeline of deliverables:
        - Wed: model definition, GUI, Cloud Infrastructure, Auth, Course info
        - Thu: minimum viable product
        - Fri: testing
    - Testing material:
        - Class RNR355 material is going to be used for testing
        - vs ChatGPT4:
        - Known limitation using openAI: max 10 files, 25MB per file, USD20 subscription
    - Thoughts:
        - Cost
        - Data ingenstion (mathematics, pdfs, images)
        - Efficiency- does every information retreival need LLMs?
        - The realities of time and effort constraints- important to focus on interpretability existing tools
        - Focus on quality of answers.

10:45

Tyson
- Winter project: set up information retreival chatbot for course materials
- React app that interfaces with OpenAI backend
- ChatGPT 4.0 chat is accurate, accuracy falls and speed increases if ChatGPT 3 is used
- Tutorbot, vs Teacherbot- future stretch goal of creating something that can work with private data between teachers and students for personalized teaching materials
- [Flowise AI](https://flowiseai.com/)  
- Could be used to create what we aim to do
- Can work with both open access and paywalled models


11:00

Nirav
- Storyboarding for the chatbot and use case, for a walk through
- Two use cases- course materials, website
- Grouper- how to restrict chatbot access to each course, so authorization is restricted
- What other universities are doing?
- Reading Math equations
- Is LLM training data private or public?
- How deep to follow URL links in a document?
- Use Local API for privacy
- Low code-no code for Professors
- Internal 
- Need to integrate other commercial solutions if plausible
- D2L is moving to Canvas (LMS chatbot)
- Office 365 (high prices for including chatbots)


Mithun
- Laying down basis for day achievable goals
- Described some initial tasks for the Team leads
- Prompt engineering 
- Each team decides more realistic tasks


1:30PM 

### Propeller Team (LLM)

- how a pipeline works.

1. Create Vector DB; Chunks of documents, pass them through LLM in order to create Vector DB.
2. Retrieval: Get the closest document to the query 
3. Generate: Depending on the task: Summary/ QA/ Translating Document etc.

Terminology:

Retrieval Augmented Generation (RAG)

- prime the LLM with context, which can help combat hallucinations
- entire text book is too long to provide in the prompt, rather we use the vector db to pick relevant chunks for dynamic prompt



*Technologies:*
- [Chroma](https://www.trychroma.com/)
- [LangChain](https://www.langchain.com/)
- [Nvidia Nemo](https://www.nvidia.com/en-us/ai-data-science/products/nemo/get-started/)



### Rudder (Back-End)
- items


### Crow's nest (GUI)
- items
- 


### Anchor (Testing/QA)
- items
- 

