# BlanketCon 22: A Post-Mortem
written by **Lemma**, **una**, **gdude**, **mango**, and **FirstnameLastname**

### TL;DR:

- Went way better than expected
- More dedicated moderators
- Distinct test/build/verify phases
- More folks able to edit site
- Fat pack
- Hotswap agent
- More time between days
- Role/permission system
- Set expectations for participants that mods can be changed/removed when they cause problems
- Whitelist mode in Bruh Moment - this needs to be done early on so people design for it
- Create/find better moderation tools - chat management, user info tools, etc
- Let's find a Discord chat integration (eg blockbot) next time
- [ModFest Utilities](https://github.com/ModFest/modfest-utilities) has a chat bridge we've used for previous fests --- Lemma
- We need the ability to mute people in our voice chat, and see who's talking at a glance

## Attendance
- 633 unique players(!)
- 170 peak concurrent players(!!!)
- Average player count 60~70 when most active, 20~30 when less active
- BTM records: 120 peak concurrent, ~470 unique

## Site Hosting

### Wunderbucket

We used this for the site this year.

+ Free...
- ...for Lemma
- Mac only
- Only one user gets access
- Slow
- Inflexible; just static files

### GitHub Pages

+ Free
+ Git-backed
+ Fast (until you hit a limit) via Fastly
+ Everyone already has a GitHub account
- Has a bandwidth limit, which we may hit with a fat pack
- Run by Microsoft, powered by Amazon
- Inflexible; just static files

### GitLab Pages

[Forum post about limits](https://forum.gitlab.com/t/what-are-the-restrictions-for-gitlab-pages-sites/15067/4)

+ Free
+ Git-backed
- Undefined "if you cause problems" bandwidth limit, but with notification
- < 10GiB repo limit
- Build time limit based on remaining CI minutes (if using their runners)
- Inflexible; just static files
~ Limits to Pages sites match those for GitLab repos

### bunny.net

> Maybe if this is determined a good solution, we (CHS) can sponsor it? We already use BunnyCDN for a couple other clients, so we could just stack into the existing bill. --- FirstnameLastname

> I also already have a BunnyCDN account for various projects of mine, and usage of it was originally my suggestion. -- unascribed

+ No bandwidth limit
+ Supports subusers
+ Independent
+ Faaaaaaaast, bunny is a CDN company
+ Hilariously flexible+unlimited "edge rules" system
- Costs money ($10/TB stored, $0.02/GB downloaded or so)
- Uploading files to it is janky; FTP works okay, SFTP is buggy, has a custom API
- No repository sync
~ Can manage files in-browser
~ Could be used to front any other hosting solution

### Vercel

We used this for pack distribution this year.

+ Syncs with GitHub repos
+ Free
+ Robust team permissions
+ Allows custom routing/etc via JSON file
- Bandwidth limit (100GB/mo)
~ Build time limit (100hr/mo - fairly generous)

### Cloudflare Pages

+ Syncs with GitHub repos
+ Free-ish
+ Can run arbitrary code on request...
- ...if you pay - otherwise no custom routing
- Cloudflare

### Netlify

+ Free and pretty fast
+ Syncs with GitHub, GitLab, BitBucket, Azure DevOps
+ Some dynamic content supported (eg forms), but with low limits on the free tier
- Bandwidth limit (100GiB/mo)
- Build time limit (5hr/mo)
- No shared ownership/control panel teams
- Ultimately still just static pages with Netlify JS injection

## Pack Distribution

### PackWiz

One of two methods this year.

- Aggressively overwrites changed/removed files
- Does not confirm updates
+ Flexible
+ Good management CLI
+ Secure (hash checks)
+ Supports download from external URLs without fallback
+ Supports optional mods, with descriptions and individual toggles
+ Can be hosted on any static site host
~ No official management GUI
~ Ugly end user UI
~ Loader version is in the manifest, but not used in practice
~ No standalone launcher
~ Installation in unsupported launchers is a manual process

### Modrinth

One of two methods this year.

- Relatively low size limit
- No way to have optional mods
+ Supports one of our "sponsors"
+ Loader version is in the manifest and is used
~ Currently experimental; we found a few bugs in launchers
~ Expects use of Modrinth, but can be put on any static site host
~ Most supported launchers cannot update (and ATL had bugs)
~ No standalone launcher (yet)
~ Relies on PackWiz in practice

### unsup

Unofficial method this year.

- Poorly supported
- Bad management interface; no complete CLI or GUI available yet
- Cannot update the loader by design
- Pet project of an organizer - potential bias
+ Robust (very careful file overwriting/etc)
+ Asks before performing updates
+ Allows changing files
+ Secure (hash checks, optional manifest signing)
+ Can be used with literally every launcher
+ Supports download from external URLs with local fallback
+ Supports "flavors", which can select between different sets of mods
+ Can be hosted on any static site host
~ Bare-minimum simplistic design - both good and bad
~ Generally considered to have a nice end user UI
~ No standalone launcher
~ Installation in non-MultiMC-based launchers is awkward

### SKCraft

We used this at BTM 16 2.0.

- Does not integrate with users' existing launchers
- Does not ask when updating
- Quilt support requires bundling a full launcher profile
- No way to have optional mods
+ Very good management GUI
+ Custom standalone launcher, can be fully branded
+ Very streamlined end-user experience
+ Can manage and update the loader as well as mods
+ Supports download from external URLs
+ Can be hosted on any static site host
~ Essentially unmaintained
~ Not designed for post-1.13 Minecraft; support is a bit of a hack
~ Usage requires forking the code

## Policies

Under what circumstances is a mod removed?
- Unfixable incompatibility
- Developer is banned for violating community rules
- Not licensed to redistribute

What kinds of patches are permitted after the pack freeze?
- Server-crashing or major TPS issues
- Exploits

## Phases

Organization was a bit of a mess; we initially underestimated time needed and had multiple people burn out. We need a more concrete system to properly set expectations for builders and keep timelines under control.

1. Testing (2 weeks)
- Messing around in a temporary world - everything will be lost
- Testing mod interactions
- Experimenting with build designs
- Letting mods get all their features prepared
- **Justification**: We had people start building on the test world, and it immediately snowballed, and testing became building.
2. Building (6 weeks)
- Building
- Pack freeze in final week
- **Justification**: People *will* put this off as long as possible - we need a fixed window so we're not seeing builds going on into the con.
3. Verifying (1 week)
- BetterThanNodus performance testing
- Profiling
- Testing for exploits in built booths
- **Justification**: We didn't have this until the literal final day. Porting BTN almost didn't happen in time, and patches almost took too long.
4. The Con® (3+ days)
5. Aftercon (2 days)
- Backup is taken before this phase begins
- Restrictions on shenanigans are lifted

## Ranks

Having all builders be operators on the final server was a disaster and a half. We need a more proper permissions system and a rank system. Pulling in a full Bukkit-esque permission node system is total overkill, though — we can just make use of vanilla's permission levels.

1. Organizer
- Permission Level 4
- Full access to everything.
2. Operator
- Permission Level 3
- Can ban/kick, but not stop the server, change others ranks, etc.
3. Builder
- Permission Level 2 - same as command blocks
- WorldEdit access, utility command access

> I noticed a couple mods were expecting us to use a proper permissions system like LuckPerms - we might need to check on that for submissions --- gdude

## Performance, Performance, Performance

Vanilla Minecraft is Very Slow. We can't avoid using a normal modded server, as that's kind of the entire point of the event. Krypton, Lithium, et al help a lot, but we still have major bottlenecks.

Packet sending is *slow* and we need to do more profiling to see which mods are sending excessive packets. Booths also need to be designed with this in mind; Automobility had an animated blackboard that was super cool but generated intense network traffic.

Many mods wrongly called `sync` on their CCA components *every tick*, which snowballed into a good 70% of our performance issues before patched. These quick patches also resulted in strife with modders, as their features were unintentionally broken by these last-ditch fixes.

However, even once we were fully patched, we could barely hold 12 TPS with 70 people online *after destroying tons of Create contraptions/etc*. Our primary bottlenecks were just in *handling the players moving around*, once all the packets were dealt with.

My best wild idea in this vein at the moment is to introduce dimension threading, exploiting the fact that we can treat the world as immutable. We can't full-on shard across servers, as Velocity is incompatible with many mods, and it would ruin the con feeling of everyone being in the same world. However, if we had multiple dimensions that are all copies of eachother on the same server, and we shadowed player entities between these dimensions, that could give the desired effect while allowing for good performance from multithreading.

Even with 170 players online, we only ever hit ~1000% CPU usage, on a box with a total max of 2400%. We have a lot of room for multithreading, if we can pull it off.

> This is somewhat incorrect as the web UI container resource monitor isn't as accurate as this shows. There were spikes up to 6 cores pegged at 100% that we saw in the historical data. --- FirstnameLastname

## Events, Scheduling, and Recording

There were a total of nine (9) hours between the last event of Day 1 and the first event of Day 2, leaving not nearly enough time for people to get sleep in between. Time zones are *very* hard to organize around, but things would have been ameliorated if the opening evening had room for panels as well. If we have a similar amount of panels for BC23, we may add on another day for them. IRL cons usually have multiple events running at once, but we think making it possible for someone to go see all of the events is something we wanna keep around.

We had multiple people recording the "traditional" way via OBS, and this went fairly well; BTMs always stopped here, but we now have solutions like ReplayMod — however, that did not work on the final server in the final pack. We should try to test this ahead of time.

## Oh God Someone Got Into Survival, and Other Exploits

A lot of mods that were part of the pack *very* much required survival mode to experiment with their booths, including Bits & Chisels, Recordable, and Jamtastic. There were a couple ways people could sneak survival mode out of these booths, as well as sneak limited items outside of booth spaces. Most notable black-market item was Hex Casting staves, which let people do all *sorts* of things to mess with people and crash the server a couple times.

A good fix for survival-mode requirements would be a sandboxing system for booths that need survival mode - potentially using Fantasy to do survival-mode booths in an instanced world. However, this can end up conflicting with the con feeling like a cohesive whole, since some booths will just need to be gateways to their sandbox dim. We may want to look into a dedicated mod for sandboxed experimentation areas for BlanketCon 23, both for survival mode and booth-specific items.

> Amy has stated that she has plans to provide gamemode-switching blocks in her mod "Peculiar Pieces", which may help some with this. --- gdude

## Applications, Updates, and Cat-Herding

We did all our booth and event applications through a Google form, which Lemma...did not realize would doxx users' email addresses if we shared the responses sheet around with edit-link mode on. We've had the big fancy idea of the ModFest Platform(tm) hovering over our heads for ages, but have never really gotten around to properly finishing it up. If we can get it done, it would be a *huge* help for managing applications and updates, since the plan for it is to be able to automagically generate our pack format based on submissions. We really should look into having a more dedicated team to work on this instead of it just being something Pyrrha tinkers with every so often.

## Moderation Woes, or the Lack Thereof

Somehow, we had...zero issues with bigotry or fashiness during the length of the con. We had a mild issue with one user talking about mature topics and making some other players uncomfy, but that was sorted out with just a message or two. The main moderation challenge we faced was [exploits](#Oh-God-Someone-Got-Into-Survival-and-Other-Exploits), which we can handle more on the preparation side for most cases. Either way, just in case some bigger problems *do* arise in future, it would be best to have a dedicated moderation team (possibly the Quilt community moderators) keeping an eye on stuff during the con instead of putting the task on the sysadmin group.

## Closing Thoughts

This con, despite all the chaos and stress we faced while getting it put together, has been an absolute delight, and we can't thank everyone involved in the con enough. We're gonna make sure it's even better next time, so see you all at BlanketCon 23!

_{and maybe sooner for a ModFest we're starting to plan 👀}

###### tags: `blanketcon` `modfest`