Staff infrastructure guide

Background: autograders

CS1660's autograders are composed of two components:

• A "loader": which is a tiny bit of code that we give to Gradescope
• The "main autograder code", which is stored in the ag-gradescope repository. Gradescope clones the main branch of this repository each time the autograder is run.

There are two important things to know about this architecture:

• To update the autograder, you generally only need to push code to the main autograder repo
• The main autograder repo contains a configuration file which tells it about which code to run for each Gradescope assignment. This guide explains how to update this while.

For more general information about the autograder configuration, see the README here and the documentation linked from it.

Most of the time, you will only interact with the autograder on Gradescope. For debugging, or to develop a new autograder, you can also run the autograder manually, or via SSH on Gradescope. For instructions on how to do this, see here.

How to prepare an assignment for release

Here are instructions on how to get an assignment ready for release. This process has several parts:

1. Check the handout
2. Upload the gearup materials
3. Test the setup guide
4. Creating the Gradescope submission
5. Updating the autograder config
6. Testing the autograder
7. Checking the stencil
8. Creating the Classroom link

All steps except the last step (creating the Classroom link) are good items to do during camp.

Note: If your assignment does not have an autograder, you can skip steps 2-3.

This may seem like a lot of steps, but don't worry, you'll be able to do it quite quickly after you try it a couple of times! Also, if the assignment already exists, you should be able to reuse most of the existing configuration. However, you need to be vigilant about checking everything to make sure there are no surprises!

Step 1: Check the handout and setup guide

The first step is to start with the documents students will read first:

• Locate the assignment handout on HackMD (Project 1) or in the latex-docs repository.
• Update the semester, deadlines, and any links to forms that may refer to content to a previous semester
• Make sure that the handout is consistent with any recent changes

Step 2: Upload the gearup materials

Now that CS1660 has well-established geraups, it is recommended to link to the existing notes and recordings rather than spending time preparing and giving new ones. The recordings from previous gearups are located here. To make them available to students:

1. Go to the course Panopto library and find the folder where the lecture recordings are located
2. In the sidebar on the left, click Create > Upload media and upload the video file
3. Once the file has been uploaded, open the recording and click on the Share button in the top-right. Then, select Copy link to get a shareable recording link.
4. Add this recording link where referenced in the setup guide, and the resources listed with the assignment on the course website.

Step 3: Test the setup guide

Before release, you should try the steps in the setup guide to make sure they work on modern systems. Even if an assignment hasn't changed, OS updates and docker updates can still cause problems sometimes. To do this:

• Remove last year's stencil link (you'll create a new one by the end of this guide
• Update any deadlines, form links, or other references to semester-specific info
• Update any references to gearups (as above)
• Follow the instructions in the setup guide as if you were a student–pretend you haven't done the assignment before and try the steps. Make sure they work as expected and match the figures in the guide well enough that the student can recognize what they're doing is correct.

Step 4: Create the gradescope submission

1. Create any gradescope submissions for the assignment by duplicating them from the previous semester. To do this, go to the Assignments page in Gradescope and select Duplicate Assignment at the bottom. (If you can't see the button, make your window bigger.)
2. Go to Settings and configure the release and due dates:

• The late due date should be 3 days after the primary due date
• The only submission method checked should be "Git". Do not allow manual uploads–this can often lead to autograder problems.

4. Go to Configure Autograder and make sure that an autograder is configured–it should show the name of a zip file, probably autograder-loader.zip. This is a pre-built version of the autograder loader.

   • If you have not changed anything about the loader (eg. what packages need to be installed, compiler versions, etc.), there is no need to update the loader from last year.
   • If you need to change the autograder loader, create a new loader archive by following the instructions here, then select Replace Autograder and upload your new zip file. Then, click Update Autograder and wait for the autograder to finish building.

   When you first clone the assignment, Gradescope will need to rebuild the autograder's container image. This will take several minutes–you can check its status on the Configure Autograder page. The process happens asynchronously, so you can do other things while it's running.

5. Repeat this process for any other submissions for this assignment (eg. the all four problems for Cryptography).

Step 5: Update the gradescope environment config

Now that the assignment has been created in Gradescope, it has an assignment ID, which means you can add it to the environment config. To do this:

1. Find the assignment ID for the assignment in Gradescope. You can find this by looking in the URL bar with any assignment page open, like this:

In this example, the assignment ID is 325373.

2. In the main autograder repo, open up the autograder environment config located at config/config.json. Locate the block for the assignment you want to modify.

3. Replace the assignment ID in the configuration block with the assignment ID for the new assignment. This is how Gradescope finds the configuration. You can generally leave all other configuration parameters unchanged.

4. Repeat this process for any other assignments you are modifying right now.

5. Save the file and push your changes to the main branch.

Gradescope should now be configured to run your assignment! Proceed on to test it.

Step 6: Testing the autograder

Once you have pushed any config file changes, test the autograder by doing the following:

1. In Gradescope, open up the assignment and go to the Configure Autograder page

2. Make sure the autograder has finished building–it should say "built at …" and show a relatively recent timestamp.

3. Click Test Autograder and upload a test submission. The easiest way to do this is to upload one of our reference repositories (eg. cryptography-reference), but you can use any submission

   If you are having trouble getting the reference repo to show up in Gradescope's list, push a trivial change to the repo (like updating the readme) and try again. Gradescope seems to populate the list by fetching the most N recently-changed repos to which you have access, and might not include the reference repo if it wasn't changed recently.

4. When the autograder completes, check the test results to make sure they match what you expect. If you run into issues, you can try running the autograder locally or Debugging via SSH.

5. Repeat these steps for any other autograders required by the assignment.

If your test results look good, your Gradescope autograder should be ready, yay!

Step 7: Check the stencil

The next step is to check the assignment's stencil code. If you haven't made many changes to a stencil,

To do this:

1. Locate the stencil repository for the assignment in the main course organization. For example, the stencil for cryptography is located here.

2. On the repository's Github page (NOT your local system), check the repository and make sure it obeys the following criteria:

   • If changes were made to this assignment during camp, make sure that the stencil has been updated to match. (Check the git history for this repo and in source-main. If you're not sure, just keep going and you'll probably notice during testing).
   • All stencils repos should have a .gitignore file.

3. On your system, make a fresh clone of the repo onto your system. DO NOT use an existing repo on your system, to avoid the chances that any latent configurations or uncommitted files affect your test results!

4. Pretend to be a student and act like you're starting the project for the very first time: follow the the steps in the setup guide to try and use the repo, for example:
   • For Cryptography: try to run the stencil binaries and make sure they work as expected, make sure the stencil code builds, etc.
   • For Flag/Handin: make sure you can run the containers per the instructions in the setup guide. For Flag, don't forget to try the Bob's router container as well.

5. As you test the stencil, keep the following mind:

   • If you need to perform any extra setup steps not mentioned in the guide, STOP. This is likely a sign that something is broken, or that the setup guide needs more instructions

   • If you encounter any syntax errors when running the stencil code, this likely needs to be fixed. For any parts of the code that a student needs to fill in, make sure that the stencil has some syntactically-appropriate placeholder so the code compiles/runs (eg. raise NotImplementedError("TODO") or pass in Python).

   • Check the results of any failing tests to make sure they make sense. Indeed, it's normal for stencil tests to fail because an implementation is missing–however, a test might also fail because, eg., there's a syntax error or some other problem with the stencil components. Look at the log/trace for any test failures and make sure it matches what you expect.
     Examples for how to interpret test failures

     Here are some examples:
     - An error like Assertion failed: expected 42.5 but result was 0 probably occurred because the function the student is supposed to fill in isn't implemented. This is normal and expected.
     - An error like cannot find symbol or Type Error: expected type X but got Y probably means that one or more files in the stencil didn't get updated. Look at these closely, compare against what you see in source-main and make sure you understand why they are happening.

6. If you needed to make any changes while completing the above steps, push your code to Github and repeat steps 3-6 to make sure everything looks good.

Once you are satisfied the stencil is correct, you can proceed to creating the Github classroom link!

Step 8: Create the Github Classroom repo and link

When you are ready to release the project, you can create the Classroom stencil link.

Important: You should wait to do this step until just before the assignment is released and you are sure that everything is ready to go. For Project 1, wait to do this until at least the end of TA camp.

Creating the Github Classroom link will make a copy of the stencil repo. If you create it early, you will need to manually migrate any changes from the stencil repo you just painstakingly checked over to the copy. This is very tedious, and it's extremely easy to introduce errors, no matter how much experience you have!

Once you are ready, do the following:

1. Go to Github Classroom and select the classroom for this semester's student org. If none exists, create one with the New Classroom button.
   • When creating the classroom, you should be able to use the default settings–do not try to add students, connect an LMS system, or try to configure anything else. We do not use most of Github Classroom's features.
2. On the classroom page, select New assignment
3. Configure the assignment like the figure below:
   • Name: Enter a short name for the assignment, like short name for the main source repo. This will be the prefix for everyone's repo name, so keep it short (eg. pr01)
   • Deadline: leave this blank. (We don't use this part of Github Classroom.)
   • Individual or Group: Select based on the assignment (Don't skip this! You can't change it later.)
   • Double-check everything, then click Continue.

4. If this is a group assignment, set the type to Group assignment and also configure the following:
   • Name your set of teams: Pick a name, like pr01-teams
   • Maximum members per teams: set this to 3 (to prepare for inevitable groups of 3)
     
5. On the next page, enter the path to the stencil repository in the source-main repo, like the figure below. The path will be of the form brown-csci0200/<stencil repo name>. After you type it, the repo should appear in a dropdown box for you to select. Some notes on this:
   • Make sure you are giving the name of the course organization (brown-csci1660)
   • If the repository name doesn't show up in the list, make check its repo settings on Github and make sure it's configured as a public, template repository. If you need to change these settings, you will probably need to cancel the stencil creation process and start over for it to show up.
   • If the repository name still doesn't show up, wait a few minutes and try again. If the repo hasn't been changed in a while, you might need to push a trivial change to the repo so that Github can find it faster. (This seems to be a known issue with Github Classroom.)

5. In the next section, configure the stencil as follows (also shown in the figure):

   • Visibility: set to private
   • Grant students admin access: DO NOT check this–make sure it is unchecked
   • Copy default branch only: check this

6. Before continuing, double-check that the repository visibility is set to private. For some reason, this setting seems to switch back to "public" when changing some other settings, so make sure it is correct!

7. When you are done, click Continue.

8. On the next page ("Grading and feedback"), ignore all of the settings and click Create assignment. (We don't use any of the features on this page.)

   After the stencil is created, you should see a page like this:

   

   While this page does have your stencil link, there's one more thing to configure:

9. At the top of the page is a line "Starter code from", and some super-long repo name. Click this repo link as shown in the figure. This should open a new Github repo (which we'll call the "Classroom stencil repo" or "copy repo", which will look like this:

   

   If this is your first time creating a stencil, take a moment to note some important things about the copy repo:

   • This repo is a copy of the original stencil repo When a student clicks a stencil link to create a repo, the code is copied from here.
   • If you need to modify code that will go to students, you will need to make changes to the copy repo. For details, see here.

10. Next, go back to the Github Classroom page (from step 8). There is one more thing to test: copy the stencil link (the one at classroom.github.com, and open it in a new tab.

11. When opening the stencil link, you should see a prompt to accept the assignment, just like a student. Click Acccept this assignment and wait for Github Classroom to create your repo. Click on the repo and make sure it looks okay.

If the repo looks good, your stencil should be ready! Congrats!

Now, you're ready to copy the stencil link into the assignment handout. If you need to make any changes, see the next section.

Modifying an already-created stencil link

If you need to alter the stencil code after a stencil link has been created, here are some things to consider:

• If the Classroom link has not been released (eg. no one has cloned it but you), it may be easiest to delete the assignment on Github Classroom, delete the copy repo (eg. pr01-stencil-s25), and create the classroom link again
• If some students have already accepted the assignment, you can't delete the existing stencil link. Instead, do the following:

  • Clone the copy repo to your system, make the changes there, and push them. Any student who clones the stencil after you push the changes will get the new version.

  • Any students who have already cloned the repo will need to make the changes in their own repo manually. To handle this, you will need to post an announcement on EdStem with instructions on how to do this. Be sure to pin the post, and to check the box to email everyone. For an example of what the post should look like, check last year's EdStem for examples (or look for examples from your other classes.)

    There's a button labeled "sync fork", can I use that?

    Unfortunately, this won't work. Github Classroom tried to make this process easy, but it only works under certain conditions that don't match how we use Github Classroom. Specifically, it requires that all students are members in the per-semester Github org, which is something we would need to set up. (Instead, we don't add students to the org and they become "external collaborators".) If you have a massive stencil change and need to use this feature, you can try doing this, but know that it won't work by default.

Super important: If you modify a stencil's copy repo, be sure to modify the primary stencil repo (ie, the one in cryptography-stencil) too! This means you will need to manually make the same changes in two repos.

If you don't do this ASAP, you will forget, the next semester's course staff won't know that you made changes (because they won't know to check your copy repo), and they'll be doomed to repeat whatever happened again! Please do this–next year's staff will thank you.

Running the autograder

There are two ways to run the autograder code. You should be familiar with both.

Running locally

To test autograder code before releasing an assignment, or to debug problematic submissions, you can run the autograder on your own system. When run in this mode, the autograder doesn't download the code from git–instead, it just reads the code and tests from your own local copy of the main source repo. This means you can quickly make changes to the code and tests and see how the autograder will respond.

To run the autograder locally, see the README in the main autograder repo.

Debugging via SSH

You can manually run and inspect how the autograder runs directly on Gradescope's cloud servers by using the Debug via SSH button in the Gradescope web interface. This is useful for debugging or testing a student's broken submission, but not suitable for long-term development.

See this page for instructions on how to debug a submission via SSH. Once you log in, you would run the script ./run_autograder to start the process. After the autograder finishes, the results should be in /autograder/results/results.json.

When you are connected to gradescope via SSH, you can browse the autograder filesystem just like you would any other linux system (cd, ls, …). You can modify files to test out certain changes, but the changes will not be saved after you exit.