Try   HackMD

Chasm Meetings

May 7th 2022 8AM (UTC)

Agenda

What happened since the last meeting?

What's the current status?

  • Chasm exists and is functional
    • Missing Issue: Can't add classes
      • Can't slice into root node because of LazyClassNode ugliness
      • Suggestion: Abstract lazy nodes (maybe reuse for methods) and clean up CoW
    • Issue: Local Variable Representation
      • Local variable indices exist, but are not useful for Chasm
      • There are multiple ways to represent locals
      • Parameters are part of locals, but: Parameters are also part of the descriptor
        • We have to ensure no redundant data in the tree
      • What about signatures?
        • Ignored by JVM, used for decompilation and reflection
        • Ideally keep them valid
        • But they contain redundant information
    • Missing issue: Records?
  • Chasm-Lang exsits and is functional
    • Issue: Intrinsics
    • Issue: Specification
    • Issue: Better Errors
  • Chasm-Gradle-Plugin exists and is functional (kinda)
    • Simple implementation, but it works
    • IDE integration is meh
      • Attached as files rather than maven dependencies (maybe ok?)
      • Attached once per source set (e.g. once for main and once for test)
      • Doesn't run on sync (maybe ok?)
    • Slow!
      • Mostly because of deflate and inflate
      • Consider "caching" of unzipped input files
      • Consider writing only files changed since last transformation
      • Speed is not super important in prod (because caching)
      • Speed is important in dev (because no caching)
  • Chasm-Access-Widener exists locally and is partly functional
    • Doesn't have descriptor matching because of the above issues
  • Quilt loader integration is missing
    • Waiting on loader merging upstream
    • Might require significant changes to KnotClassLoader
  • Chasm-Mixin is missing

What needs to be done next?

  • Focus on Chasm class issue
  • Focus on Lang Specification
  • Focus on Loader
  • Focus on Accesswidener
  • After that:
    • Development branch
    • Repository versioning

Additional Notes

  • Meetings every 2 weeks (in the weeks without dev meeting)
  • Somewhat fixed time: Monday 5pm UTC before dev-meeting
  • This is just a test

May 16th 2022 5PM (UTC)

Agenda

  • Remapping

What happened since the last meeting?

Nothing

What's the current status?

Same as last meeting

What needs to be done next?

Same as last meeting

Remapping

Mixin

  • Mixin is based of Strings
  • Mixin annotations don't get remapped.
  • Refmap is gnererated containing mappings from dev names to runtime names
  • Annotations are remapped at runtime

Chasm

  • We need to remap transformers
  • Preferably at compile time, but needs to be supported at runtime
  • Different ways of doing it:
    • Regex replace in the actual transformer
      • Make "special strings" that get remapped
      • Pro: Doesn't require much work work in chasm
      • Con: Only works for fully qualified names
    • Make a special node in the chasm structure
      • E.g. "remap" object in the root node
      • Pro: No extra logic in chasm
      • Con: Remapper needs to understand chasm file format
    • Intrinsic function for remapping
      • E.g. remap("net/minecraft/World")
      • Con: Remapping only at runtime
      • Con: Frontend needs to be mapping aware
      • Pro: No extra work in chasm required (intrinsics provided by runtime)
      • Pro: Remapper doesn't need to understand chasm file format
    • Extract special node to a separate File
      • E.g. "remap" object in separate file
      • Pro: Remapper only needs to remap json
      • Con: Special logic to import external files BUT wanted anyway (for config or similar)
    • Put references into special class containting references
      • E.g. transformerId.Remap
      • Pro: Remapper needs no extra work
      • Pro: No extra work in chasm
      • Con: Cursed af
      • Con: Bigger files
  • Solution number 4 seems preferred

The transformer (doesn't need to be remapped):

// transformer.json
{
  id: "ExampleTransformer",
  remap: file("refmap.json"),
  trasformations: [
    {
      target_class: classes[c.name = $.remap.classes.world]
    }
  ]
}

Refmap in dev, generated by frontend:

// refmap.json
{
  mappings: "org.quiltmc:quilt-mappings",
  classes: {
    class_0: {
      name: "net/minecraft/World"
    }
  },
  methods: {
    method_0: {
      class: "net/minecraft/World",
      name: "methodName",
      argTypes: [
        "Lnet/minecraft/entities//Entity;"
      ],
      returnType: "Ljaval/lang/String;"
      descriptor: "net/minecraft/World/methodName(Z)V"
    }
  }
}

Refmap in mod, remapped by build tool:

// refmap.json
{
  mappings: "org.quiltmc:hashed",
  classes: {
    class_0: {
      name: "net/minecraft/unmapped/C_25nhd73"
    }
  },
  methods: {
    method_0: {
      class: "net/minecraft/unmapped/C_25nhd73",
      name: "m_fhjd73n",
      argTypes: [
        "c_hgd9dud"
      ],
      returnType: "Ljaval/lang/String;"
      descriptor: "net/minecraft/unmapped/C_25nhd73/m_fhjd73n(Z)V"
    }
  }
}

May 30th 2022 5PM (UTC)

Agenda

  • Additional files/includes
  • Versioning
  • Blanketcon

What happened since the last meeting?

What's the current status?

  • Chasm exists and is functional
    • Issue: Local Variable Representation
      • See above
    • Missing issue: Records?
  • Chasm-Lang exsits and is functional
    • Issue: Intrinsics
    • Issue: Specification
    • Issue: Better Errors
    • Issue: Treat classes as a fucntion input
  • Chasm-Gradle-Plugin exists and is functional (kinda)
    • Simple implementation, but it works
    • IDE integration is meh
      • see above
    • Slow!
      • see above
    • Doesn't work with loom
  • Chasm-Access-Widener exists locally and is partly functional
    • Doesn't have descriptor matching because of the above issues
  • Quilt loader integration is missing
    • Waiting on loader merging upstream
      • Loader is diverging from upstream
    • Might require significant changes to KnotClassLoader
      • Probably replace the entirety of Knot
      • Wait for loader-plugins since that also messes with Knot
  • Chasm-Mixin is missing

Additional Files/Includes

We want to include additional files in transformers, e.g.

{
  id: "example_transformer",
  refmap: include("refmap.json"),
}

Problem: In order to allow caching, we need to be aware of all used input files.

Solution:

  • Keep track of files used during last execution, rerun if any changed
    • Requires some additional logic in chasm to emit file list
  • Pessimistically assume all files in a certain path to be required
    • No extra logic in chasm, but might hash unnecessary files
    • Disadvantage: "magic direcory" makes it hard to differnetiate between transformers and libs

We choose option one.

Versioning

Currently: Chasm uses 0.0.1-SNAPSHOT for everything.

Once chasm becomes "somewhat stable", we need better versions. Suggest a git layout for the "far" future:

  • Main branch
    • Always the lastest release version (e.g. 1.2.3)
    • Bump major opr minor version when merging development branch
  • Development branch
    • Always has the next minor release snapshot version (e.g. 1.3.0-SNAPSHOT)
      • Probably just released on the snapshot maven
    • Aways based on the main branch
      • Rebase when a patch gets applied
  • All development happens in PRs
    • Feature PRs
      • Target development branch
      • Merging doesn't bump the version
    • Patch PRs
      • Target main branch
      • Bump patch version

Blanketcon

Blanketcon is happening and we have a keynote June 11th, 18:00 UTC, directly after quilt dev meeting. We need to prepare a presentation!

June 20th 2022 5PM (UTC)

Cancelled

June 17th 2022 5PM (UTC)

Agenda

  • Module Restructure
  • JavaCC21
  • Lang CodeGen
  • Lang API

What happened since the last meeting?

Current status

Same as last time, except one PR for restructuring: https://github.com/QuiltMC/chasm/pull/53

Restructure Modules

Chasm and Chasm-Lang interop will be removed from Chasm-Lang to make the module dependency free. Two options:

  • Move the code into the chasm module
  • Move it into a new module For now, we choose the first option.

Switch to JavaCC21

Antlr problems:

  • Needs dependency that is 3 times the size of chasm
  • Shadowing is awkward with licensing

Switching to JavaCC: Pros:

  • Chasm-Lang is dependency free
  • Chasm-Lang is much smaller (including dependencies)
  • No licensing concerns whatsoever Cons:
  • Requires a somewhat sizable rewrite
  • JavaCC is ooooold and outdated -> JavaCC21 instead
  • JavaCC21 has no versioning
  • JavaCC21 has no docs, but it's mostly the same as JavaCC

Conclusion: Leave issue open until it bothers me.

Lang Code Generation

People have to generate Chasm-Lang. For both frontends (Mixin, AW) and the remapper. It would be nice to have a library for this. Basically:

  • Construct/Modify an Expression
  • Write an Expression to file
  • Or similar?

Lang API

Currently there's no api/internal split. Additionally, the public API isn't "nice". Needs work.

July 18th 2022 5PM (UTC)

Agenda

  • Go over PR#54
  • Discuss further steps

What happened since the last meeting?

PR 54

  • Switch to JavaCC21
    • Concerns: Bad dependency
  • AST is currently free of implementation detail
    • Bad OOP, likely to change it back
  • Currying is broken
    • Probably best fixed with closures
  • Consider replacing the Chasm specific tree with the new Chasm-Lang AST
    • Saves on conversions
    • Reduces code base size
    • Possibly a lot of work

August 15th 2022 5pm 6:30pm

Present:

  • CheaterCodes
  • Kropp

Agenda

  • Review current stauts
  • Contribution Guideline
  • Chasm Roadmap
  • Innclude directives
  • Json parity
  • Parser Talk

What happened since the last meeting?

PRs:

  • #54 Lang rework (by CheaterCodes)
  • #55 Chasm Renderer (by RTTV)
  • #57 Map & Flatten (by will)
  • #58 Fix Annotations (by FoundationGames)
  • #64 Add Metadata (by CheaterCodes)
  • #65 Fix Slice Offset (by CheaterCodes)
  • #66 Fix Trailing Commas (by CheaterCodes)
  • #67 Source Spans (by CheaterCodes)

Lots of testing by FoundationGames:

  • Add chasm to nil-loader (kind of)
  • Start on a mixin fork (Chasmix)

Currently in the works

PRs:

  • #61 Fix operator precedence (and more) (by Kropp)
  • #60 Fix error in writing annotations (by Kropp)
  • #59 Map operators (by FoundationGames

Contribution Guideline #50

  • CONTRIBUTING.MD
  • General stuff, PR procedure etc.
  • Every bugfix PR should included tests

Branch protection is enabled:

  • Contribution only via PR
  • Requires one Review (from the team)
  • Must pass checks (build + tests + checkstyle, aka ./gradlew build)
  • Must be up to date (via merge)

Chasm Roadmap

TLDR: I want to write stuff down without creating issues.

GitHub Projects?

  • Would allow for tracking milestones
  • Allows short entries without issues
  • Also tracks issues
  • Extra effort for no gain?

Include directives

We want the option to include other Chassembly files in a Chassembly file. E.g.

{
  lib: include('other/lib.chasm'),
  call: lib.helper(5),
  ...
}

Problem: Who's managing the filesystem?

  • Easy solution: Don't provide this functionality
    • Execution environment can provide an intrinsic (e.g. quilt-loader)
    • Problem: Not portable
    • Problem: Makes Chasm harder to use
  • Better Solution: Require an implementation
    • Execution environment must provide an intrinsic (e.g. quilt-loader)
    • Problem: Makes Chasm harder to use
  • Big change: Make Chasm filesystem aware
    • Move all jar unpacking logic etc. into chasm
    • Problem: More work for Chasm, possibly less flexibility
    • Problem: Even less portable?
  • Best Solution?: Require an implementation, provide a default in ChasmUtils
    • Chasm provides an optional default implementation, but allows replacing it.
    • Problem: Possibly complex/out of scope

Json Parity

Include is intended for Chassembly files. Main usecase: Reference (remapped) class names in separate file.

Question: Should those files be Json? Should Chassembly be a superset of Json?

Reasoning: Remappers might want to use a Json library and not Chassembly. Specifically, "new quilt-mappings" might be Json based, so Json library is already bundled.

What's needed to read Json?

  • Strings as keys
  • Number formatting?
  • Parsing details? (e.g. String escapes)
  • Do we want to test this? (test suite)
  • How much is missing for json5?
  • How much is missing for js?

Parser Talk

Previously: Antlr4

  • Adaptive LL(*)
  • Visitor Based
  • Priblem: Big runtime dependency (>300kB)

Currently: JavaCC21

  • LL(k)
  • Embedded Java Code based
  • Problem: Lots of code repetition

Alternatively: JavaCC21 with tree building #63

  • LL(k)
  • Autogenerate Parse Tree
  • Problem: Generated AST is almost useless
    • Nodes are mostly just empty marker classes
    • Functionality needs to be injected
    • 50kB of empty classes
    • Not really suitable as API

Ideal: Parser generator XX

  • LL or LALR or whatever, powerful enough for Chassembly
  • Visitor Based (No parse tree required)
  • No runtime dependency, small footprint
  • Problem: Don't know any

Good news: Parsing is not an implementation detail

  • Public AST should remain stable
  • Changing the parser shouldn't require a full rewrite

September 14th 2022 5:30pm

Present:

  • Kropp
  • Earthcomputer
  • CheaterCodes

Agenda

  • Review current stauts
  • Include Directives
  • Release Roadmap
  • Labels
  • Chassembly Optimization
  • Chassembly IntelliJ Plugin
  • ASM Chasm Frontend?

What happened since the last meeting?

  • Multiple bugfix PRs #70, #71, #72, #74
  • Include Intrinsics #73

Include Intrinsics

  • New intrinsics to include files:
    • file
  • Implementer must provide a readFile function:
    • Must be pure
    • Path format (currently) implementation defined
      • Must be OS independent
      • Possibly also allows configs e.g. include("config:modid/config/name")

Release Roadmap

Github project now available under https://github.com/orgs/QuiltMC/projects/12/views/1

  • Every team member can (and should!) add items to the list
  • Every team member can (and should!) convert drafts to issues
  • Please use liberally!

Labels

Problem: Copying instructions from one method to another poroduces invalid bytecode. How should this be handled?

  • Transformers SHOULD prefix labels they produce with their ID
  • Method labels are not globally unique
    • Method labels could be unique, but this doesn't actually solve problems
    • Two transformers copying from the same method still need to prefix
    • Prefixing isn't hard
  • Not enforced in code?
    • Not sure if we could
    • General assumption:
      • Compiler developers know what they're doing
      • So that users don't have to
    • Needs documentation!!
  • Transformer ID is provided by execution environment
    • We need to pass the ID to the transformer (just like classes)

Optimization

Why is it needed?

  • Recursion is essential to transformers
    • E.g. Looping through a list
    • Intrinsics only help for (common) special cases
    • Default performance is suboptimal

Possible optimization:

  • Common subexpression extraction and reordering
    • Must maintain original behaviour (especially errors)
  • Tail recursion
    • Reorder for more tail recursion
    • Perform function substitution late to keep stack space small
  • Mutability
    • Lists are immutable
    • Modifying a list requires copying it
    • If the original isn't used afterwards, the list can be mutated instead
    • Similar to Rust move vs. copy

Chassembly IntelliJ Plugin

Earthcomputer has been working on an IntelliJ plugin for Chassembly language support.

  • "Not meant to help writing Chassembly by hand!!1!11!" (exceptions apply)
  • Make frontend development easier
  • Mostly for writing libraries and for manual experimentation
  • Not yet published

ASM Chasm Frontend

Earthcomputer has been working on a Chassembly JVM bytecode interpreter

  • Support legacy ASM trasformers
    • Partially suppoirt mixin plugins?
  • Limitations apply
    • Still pure (e.g. no internet access)

Kotlin Frontend?

Kotlin team has created a channel for discussing frontend ideas. No specifics yet.

January 11th 2023 20:00pm UTC

Present:

  • Kropp
  • Pyrofab
  • Will
  • Earthcomputer
  • CheaterCodes

Agenda

  • Chasm pre-release
  • Transformer Format
  • Lock Lifetimes
  • Transformer Sorting

What happened since the last meeting?

  • Lots of Chassembly things (syntax, intrinsics)
  • "Virtual" Transformers

Chasm pre-release

Last changed by 
 
CheaterCodes
0
1
259
Published on HackMD