Deep Learning Final Project

The final research project is aimed to give you an idea of what a deep learning research project entails, and hopefully, get you excited about doing research in this field. It requires critical thinking that you will develop by learning the material and doing assignments during the semester.
At the end of the semester, you will share your work with your peers through presentations and writing a report. This type of project-based exercise will help you in developing skills to perform independent research like:

• Thinking of a project idea
• Doing literature survey
• Designing experiments
• Reporting results in a clear and concise manner.

Overview

Students will complete final projects in groups of 2-4 people. These projects are an opportunity for you to apply the knowledge you’ve gained in class to a topic or area of interest. Your group will create its own novel project. The time commitment per student should be approximately equal regardless of the group size, thus larger groups should justify their group size by taking on more ambitious projects.

You will be graded on completion of goals, punctuality with which you meet your deadlines, and professionalism in reports and presentations.

We do not expect you to build a magical model that solves all of the world’s problems. What we do expect is concentrated and well thought out effort into solving a problem: Rather than reporting a single number using a certain metric and showing your model just “works”, we expect you to perform quantitative ablation studies (e.g. show us what architectural changes, hyper parameters, and regularization techniques you tried and what are helpful, explain why you think so) to analyze your models, along with qualitative evaluation (e.g. visualizations) to illustrate what the model learns and how it might fail. You might want to check Prof. Bill Freeman’s awesome talk on “How to Write a Good CVPR Paper” (video, slides). Of course, we do not expect you to finish a project at “top AI conference” level within a semester, but the general high-level principles are helpful.

Do not be afraid of negative results! Some of the most interesting and well-received presentations from past years were ones that failed to produce “successful” results!

Requirements

• Groups of 2 to 4 students (A solo project is possible, but not recommende. The student should check with the instructor and get approval.)
• The project must be related to course material.
• Present and submit a slide deck of the project at the end of the semester recapping your work.
• We will host a presentation day on December 10th. Student groups will participate in-person for presentations.
• Submit all project-related code via sharing a Github Repo link. Projects without code submission are considered incomplete and will not be graded.
• Your code can be written using any deep learning library, such as TensorFlow, Keras, Jax, or PyTorch. We recommend you take on a project that requires moderate-level of compute (e.g. your desktop, or department machines).
• Reports and proposals must be written as PDF files.

Scope

The project is “open-ended,” meaning it’s open to interpretation. Above all, you should do a project that you love! We expect your team to apply what you have learned from the course to solve a new problem, concretely:

You can do this using whatever deep learning methods you can find that get the job done. Ideally, the project would involve more than one major topic we covered in the class (CNNs, RNNs or Transformers, Generative models, Fairness and model interpretability, and Reinforcement Learning). You are encouraged to implement your project “from scratch”, but can also use open-source deep learning projects as a component in your framework. Example use of open-source projects that are permitted:

• You are building a generative model that turns food videos into recipes, you can extract visual features with an open-sourced model, such as CLIP.
• You want to build a reinforcement learning model for a game you invented, you can try out some well implemented RL baselines.
• You run a thorough analysis of model bias (e.g. gender, racial) for one or a collection of popular deep learning models that have been open-sourced.

Example uses that are not permitted:

• You take open-sourced model checkpoints and just “fine-tune” them on another dataset.
• You take an open-sourced framework and replace its ResNet-50 with a ResNet-101.

In short, by "new" we focus on the creativity you demonstrate in the project (e.g. you don't need to artificially create a problem that nobody worked on before) and the amount of work required to realize your idea (namely, your model design doesn't need to be "novel" for its own sake. Implementing an existing technique that is not covered in the lectures, but is natural to be applied for your problem, and brings valuable insights, is greatly appreciated.)

Please cite all the open-source frameworks you used in the final report, and check with your mentor TA if they are okay with your proposal.

You can check out the projects in the previous Deep Learning Day (2021, 2022, 2023), CSCI 2952-N or Stanford CS231n to draw some inspiration, but your project needs to differentiate from previous projects in meaningful ways. Please note that some of the Deep Learning Day projects may not meet our required scope, since they were for 1470 students.

Compute Resources

We recommend a project that can run on your desktop or department machines. Please check with the course staff if your project proposal might require excessive compute resources than we can provide. Remember that you will not be judged for the absolute performance (e.g. you get the best numbers in the world on some benchmark), but the creativity of ideas, quality of code and documents, as well as thoroughness of the ablation experiments.

If you really want to take on a very ambitious project that requires several GPUs with large memories, please ensure that you have obtained the computational resources before you start with the project. Example resources:

• You are working in a research lab and the lab provides you with GPU machines.
• A free exploratory account from Brown CCV (choose the “exploratory account without a PI” option).
• Try Cloud service providers that offer free student credits (e.g. GCP). Google colab notebooks will also let you use 1 GPU for free.

A Note on GenAI APIs

As you will experience in mini project #2, it becomes increasingly easy to utilize powerful deep learning models in your project by importing them from huggingface or calling their public APIs. You may use these models in your project after consulting with the instructors (e.g. on Ed or via email). Since these models operate as big black boxes with natural language as interface, the instructors would like to ensure that your proposed project still validates your knowledge on deep learning, as opposed to being a software engineering oriented project.

Deliverables

Project Proposal

With your team members, decide a team name, and submit your final project idea by filling out the form here! Only one person from your group needs to submit the form for everyone. If you are trying to solve something new, please describe the problem and your plan of action. We will approve all that are appropriate.

The purpose of having your project description is to provide a quick feedback on your project scope and its feasibility. You may (ideally after consulting the TAs or the instructors) change the project topic, untill the project check-in.

Project Check-in

You will write a one-page reflection on your progress so far. We expect you are wrapping up the implementation and performing final experiments. If you have questions, please contact the course staff, or post questions on Ed. You'll submit this reflection on gradescope.

For this checkin, we also require you to write up a reflection including the following:

• Introduction: What is the problem you try to solve, why do you pick this topic?
• Challenges: What has been the hardest part of the project you’ve encountered so far?
• Insights: Are there any concrete results you can show at this point?
  • How is your model performing compared with expectations?
• Plan: Are you on track with your project?
  • What do you need to dedicate more time to?
  • What are you thinking of changing, if anything?

Deep Learning Day

This is a chance to show off your teams awesome project and see all of the great work your peers have done! More logistical details about the event and participation will be shared as we get closer to the event.

Students should prepare a 5-minute presentations describing their project using slides.

Your slides must contain the following information:

• Title
• Names of project group members
• Introduction: what problem you’re solving and why it’s important
• Methodology: your dataset and model architecture, etc.
• Results: both qualitative and quantitative (e.g. if you’re doing an image-related project, we want to see both pictures and graphs/tables)
• Discussion: lessons learned, lingering problems/limitations with your implementation, future work (i.e. how you, or someone else, might build on what you’ve done)

Final Projects Due

You will need three final deliverables submitted through gradescope (note this is a hard deadline):

1. Your presentation slides/content
2. Finalized code on GitHub (provide the link in a README)
3. Final writeup/reflection

Final Writeup/Reflection

Along with your Devpost submission, provide a final write up/reflection of the final project.

Please be sure to include:

• Title
• Who
• Introduction
• Related Work
• Data
• Methodology
• Results
• Challenges
• Reflection

Your Final Report should contain the following:

• Title: Summarizes the main idea of your project.
• Who: Names and logins of all your group members.
• Introduction: What problem are you trying to solve and why?
  • Detail how you arrived at this topic and what motivated you.
  • What kind of problem is this? Classification? Regression? Structured prediction? Reinforcement Learning? Unsupervised Learning? etc.
• Related Work: Are you aware of any, or is there any prior work that you drew on to do your project?
  • Please read and briefly summarize (no more than one paragraph) at least one paper/article/blog relevant to your project.
• Data: What data are you using (if any)?
  • If you’re using a standard dataset (e.g. MNIST), you can just mention that briefly. Otherwise, say something more about where your data come from (especially if there’s anything interesting about how you will gather it).
  • How big is it? What kind of preprocessing was required?
• Methodology: What is the architecture of your model?
  • How are you training the model?
  • Justify your design. Also note some things you tried that may or may not have worked.
• Results: What constitutes “success?”
  • What experiments did you run?
  • For most of our assignments, we have looked at the accuracy of the model. Does the notion of “accuracy” apply for your project, or is some other metric more appropriate?
  • Explain how you assess your model's performance and justify why it is a reasonable measurement.
• Ethics: Choose 2 of the following bullet points to discuss; not all questions will be relevant to all projects so try to pick questions where there’s interesting engagement with your project. (Remember that there’s not necessarily an ethical/unethical binary; rather, we want to encourage you to think critically about your problem setup.)
  • What broader societal issues are relevant to your chosen problem space?
  • Why is Deep Learning a good approach to this problem?
  • What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?
  • Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?
  • How are you planning to quantify or measure error or success? What implications does your quantification have?
  • Add your own: if there is an issue about your algorithm you would like to discuss or explain further, feel free to do so.
• Division of labor: Briefly outline who will be responsible for which part(s) of the project.
• Reflection: please address the following questions (along with other thoughts you have about the project, if any):
  • How do you feel your project ultimately turned out? How did you do relative to your base/target/stretch goals?
  • Did your model work out the way you expected it to?
  • How did your approach change over time? What kind of pivots did you make, if any? Would you have done differently if you could do your project over again?
  • What do you think you can further improve on if you had more time?
  • What are your biggest takeaways from this project/what did you learn?

Grade Breakdown

A project submission is considered complete if the written report, presentation, AND code are all submitted. An incomplete project receives zero grade. A complete project will be graded as:

• Written reports: 30%
• Oral Presentation + Slides: 30%
• Code: 30%
• DL Day Participation: 10%

(Special permission might be granted by the instructor to students who have to attend the deep learning day remotely, without being penalized for the participation grade.)