10'000 m goals

1. Make the demoscene more accessible to people who may have never heard about it.
   (
   Image Not Showing Possible Reasons

   • The image file may be corrupted
   • The server hosting the image is unavailable
   • The image path is incorrect
   • The image format is not supported

   Learn More →
   hello #GLAMhack!)
2. Create an open dataset of demoscene productions, which could be filtered to individual themes or platforms.
3. Support the UNESCO digital heritage application, or just explore the history of the 'scene.

..Or just tune into SceneSat

Elevated by Urszula "Urssa" Kocol

Our GLAMhack roadmap

• 10 Mine some data from demozoo
• 20 Collect a few bytes from pouët
• 30 Create an initial Data Package
• 40 Document process of aggregation
• 50 Set up a basic demo service
• 60 Push dataset as open repository
• 70 Propose a standard(TM) schema
• 80 Create and demo the prototype
• …
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

The drilldown

Road by PG and R0ger

Pouet logo by tomaes

trumpets

The oldest and most well known central repository, pouët, makes daily data exports available at https://data.pouet.net/ with a JSON API endpoint at https://api.pouet.net/ and the open source code of it at https://github.com/pouetnet/pouet2.0-api

We downloaded and tried to parse the raw JSON with a couple of tools, and didn't manage to get far. Convoluted structure and formatting errors were rather demotivating. Nevertheless it influenced our thoughts about a "demoscene data standard", and brainstorm ideas of improving overall data quality (for example, we immediately noticed mismatched dates and missing values).

This API was also recently used to do some terrific data analysis, and we reached out to the authors to find out if we can reuse their scripts. We have also reached out via the #pouet Discord for guru meditation.

demozoo

The Demozoo API is basic but usable: http://demozoo.org/api/v1/
As it's a paginated web service, it would require a bit of scraping code to aggregate. So we used the 'nuclear option' of getting the database dump in raw SQL format. Importing this into a local SQLite database (inspired by all of Wikipedia in SQLite) and then re-exported the tables in CSV format. This should be done differently for automated data updates.

We have reached out via the #demozoo Discord and GitHub for some further ideas.

package

A popular current way to crowdsource open data in a distributed way is the Data Package, the preferred format for doing this using the Frictionless Data project, which has a create tool to generate an initial datapackage.json.

An initial data package based on the Demozoo archive is at GitHub, which compiles and aggregates the data from several tables using the Python dataflows library.

schema

Exploring and transforming the data gives us a frame of reference based on which to think about some commonalities and differences between different archives' approaches. We did some research to see what effort in this direction was already made, and reached out to the Demozoo and Pouet communities.

Each of the data sources have a schema of their own, and some attempts at consolidation have been made. We started with a simplified version of the Demozoo model, created a Table Schema which can be used for validation or annotation as JSON Schema.

repository

Our dataset repository clearly explains its sources, but also points out there are many other places which could be future data acquisition targets. These notable scene repositories and data sources include:

• https://demozoo.org/ (as above)
• https://pouet.net/ (as above)
• https://csdb.dk/
• https://zxart.ee/
• https://ada.untergrund.net/
• https://files.scene.org/
• https://www.demoparty.net/

service

While the Data Package is nice to look at in it's JSON glory, most people (and programmers) will want some kind of interface to it. We wrote a small server using the Falcon Framework to produce a barebones API. Since the data is loaded using a Frictionless Data wrapper for the Pandas library, it can incorporate various advanced sorting and filtering routines. Our proto-service is currently running at a private VPS hosted on Linode, but should also work on 'lambda function' hosts like Vercel or Heroku.

Still from Traffic Jam by Chainsaw

prototype

If we make it this far, we would like to make a basic example of data usage. After all this data wrangling, we didn't have time to really explore the space of user interface possibilities. But we have a small application that demonstrates the API with an infinite-scrolling user interface showing productions.

See for yourself at https://scene.rip

Ideas to build upon

• Establish live feed graph view of demoscene productions
• Create an infographic that helps to explain the demoscene
• An interactive app for scrolling through prods (e.g. Netflix or Giphy clone)
• A cheatsheet to learn the most important terms and famous groups/prods
• A virtual reality exhibition (like other teams are working on)

More or less irrelevant links

• Thread showing various graphs of prod release stats from Pouët https://twitter.com/pouetdotnet/status/1345056403338231808
• Discussion of "greets graph" https://www.pouet.net/topic.php?which=12099&page=1
• Some debate about classification https://www.pouet.net/topic.php?which=12098
• Discussion of Demozoo API https://demozoo.org/forums/66/
• Oleg's blog explaining the demoscene, partially on the topic of "letting the data speak for itself" https://blog.datalets.ch/010/
• Teaser Revision 2017 seminar "Graph databases and the demoscene universe" https://2017.revision-party.net/events/seminars
• Ideas from demosceners on GitHub https://github.com/nesbox/TIC-80/discussions/1286
• Internet Archive gallery https://archive.org/details/softwarelibrary_c64_demos "To some, the heart of the Demoscene - the self-playing examples of programming and artistic prowess of the last 30 years on the underpowered but extremely flexible C64."

Demo or

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

die!

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

Thanks for watching!

Pass it forward: hackmd.io/@oleg/the-scene-lives

This presentation is shared under CC BY 4.0