# CS50 Evaluation Report

Since our [workshop](https://github.com/github/runtime/discussions/611) at Harvard in December, students of the Computer Science class (CS50) were able to use Runtime to deploy their final assignments and utilize a PostgreSQL database for a production and preview environment per repository for the period of four weeks. We have been conducting multiple customer sessions and sent students a survey before closing the test phase on January 9th.

Students were able to use Runtime to deploy 43 projects, some key figures:
- 34 used Python with the web framework Flask
- 4 were in Python only
- 3 in NodeJS
- 1 in Rust
- 1 in Perl
- 35 used Runtime
- 31 had a successful deployment
- 75 succeeded workflow runs, 37 failed, 4 cancelled
- 26 required a database
- 16 created and used a PostgreSQL database
- 1 used a third-party MongoDB

This post just scratches the surface of the information we have been gathering during testing. Please see the detailed **[user research study](https://docs.google.com/presentation/d/1BEKhLo7hIIwQKL_FrOzByJJQqs8DJRrRa0qJCMWlF4g/edit#slide=id.g1ce5b45f055_0_112)** from @jnstuart for more granular information.

## Key Learnings
- A critical feature gap for students was the lack of access to container startup errors. It’s extremely important for less experienced developers that don’t have a good understanding between local and production development. They are used and taught to do everything in the cloud, e.g. using Codespaces, with several helpers active like commit/push on file save. The consequence is that they expect a production environment to be similarly easy to control and debuggable. We are surfacing the startup error logs on the CLI in the meantime and look forward to doing it via the UI as well.
- Deployments related content isn’t discoverable and users didn’t expect it to be under Actions, as confirmed in early customer sessions with students. Surfacing both application builds and deployments prominently and cohesively is important.
- Overall, the most requested feature has been to support custom domain names. Which we are aware of to be a showstopper for Runtime to be used for anything other than just testing purposes. It’s one of our main focuses to accomplish during Q3.
- The main friction for students wasn’t Runtime itself but the preliminary steps due to experiment limitations: This included joining a special organization, creating a repository and importing their project, using Codespaces and the CLI to activate Runtime. Students were not experienced with Git and database migrations posing a challenge to migrate their database from their local SQLite to PostgreSQL.
- On the data side, we received extremely useful insights on actual Azure Container Apps charges. Not only can we see the individual usage but also the cost pattern and thus better project real world charges from the complex Azure Container Apps pricing structure better. The two main learnings were
- the disproportionally high cost of the Azure Container Registry (we are looking into using the GitHub Container Registry going forward)
- how the idle cost is preventing Azure Container Apps from reaching small to no-revenue customer segments like students, indie developers and small start-ups.

## Costs
- Azure Container Registry [Premium tier at $50](https://azure.microsoft.com/en-us/pricing/details/container-registry/) per owner per month (before discounts).
- This is by far our single biggest upfront spend per owner now, we’re looking to reduce this by migrating to GitHub Container Registry or changing usage of container registries to cheaper SKUs in Q3 with similar security posture.
- Each repo gets two Container Apps, one for Preview deployments and one for Production:
- Preview is rarely used, both container apps start with a welcome container. We will re-consider this as it generates avoidable costs if unused.
- Each app starts with a hello-world container, accumulating $.30 per day in idle cpu/mem cost if not used. We've opted to scale preview deployments down after an hour and only scale them up on request.

### Cost for the `runtime50` organization resource group

| Service | Plan | Price |
|--------|--------|--------|
| Azure Container Apps | Standard vCPU Idle Usage | $146 |
| Azure Container Apps | Standard vCPU Active Usage | $0.15 |
| Azure Container Apps | Standard Memory Idle Usage | $293 |
| Azure Container Apps | Standard Memory Active Usage | $0.04 |
| Azure Container Registry | Premium Registry Unit | $28`*` |
| Log Analytics | Data Ingestion | $1.23 |
| Average cost per project | | $10.89`**` |

## Misc Feedback
- Using Runtime from Codespaces has been a struggle for students since the repo scoped token won’t come with the right permissions. The workaround is to unset the token and sign in again for `gh` to work since all Runtime operations currently require admin privileges on the repo for the token.
- When uploading files via the "Add files" functionality in a repository only drag and drop preserves the file structure (including folders) but selecting from the filesystem does not.

`*` this is per GitHub account, if every student uses their personal account, it would be $1,204
`**` the cost is without any optimization's measures on our end. Optimizations are prioritized for this quarter