Pallet repo re-structure

Immediate actionable items:

• Try to publish Substrate / Cumulus to crates.io and fixes issues that preventing it
• Move XCM out of Polkadot
• Setup publish pipeline

Future actionable items

• Start actually using semvar
• Identify bottleneck of pallet development
• Identify the most wanted pallets from parachains

Discussion

• Decide the repo structure
• Decide the development process for the new repo / github org

Considerations & Solutions

• The more repo, the more overhead, the more work required to keep them in sync, the harder to perform large refactor across multiple crates, the more likely to introduce compatibility issue.
  • Substrate is now used by many teams to the point it SHOULD be stable. We NEED it to be stable. While we want to be innovative and constantly improving the core, we should start avoid making breaking changes as it adds a lot of additional work to all the downstream projects.
  • We have a reasonable workflow to ensure cross repo compatibility via companion PR. But the fact it is needed is purly because we don't have releases. With proper release & semvar, dependency management is a solved problem.
• This is too much work. We all are busy with everything else and have no time to implement this.
  • This is blocking all the parachain teams from be able to build & collaborate pallets productively. This is a high priority task.

Proposal

• Create new Substrate Github org
• Use Github discussion for public discussion
• Use Github project for task management
• Assemable a lead committee
  • People from Parity, W3F, and parachain teams, needs to be a signficant contributor
  • Should be small to ensure efficient decision making
  • Decides what to build next
  • Triage issues and assign priority
  • Monthly calls
• Assemable a working group
  • Contribute to join
  • Need to be respoonsible for their contributed code.
    • i.e. become code owner for that
    • Need to be able to perform maintaince/review duty in timely manner
• Setup automation to ensure the code quality
  • Basically the current setup of Substrate

Pallet development lifecycle

• Proposal
  • Anyone can submit pallet ideas and they enter the backlog for futher review & discussion
• Discussion & Design
  • Refine the ideas, determine use cases, identify potential projects that will actually use this pallet
  • We should only include pallet that wanted by at least two different teams
  • Someone collect all the discussion results and create a pallet design document that includes detailed technical requirement
    • Types
    • Config
    • Storages
    • Calls
    • Hooks
    • Pallet methods
  • Needs to be reviewd
• Implementation
  • Implement the design and submit as PR
  • Pallet status will be draft indicating more breaking changes are expected and should not be used on mainnet
• Improve
  • Futher improvements and fixes
  • Can transit into experimental status indicating runtime migrations are required for storage breaking changes, and other breaking changes requires an announcement
• Audit
  • Once the pallet is audited or sufficiently tested on live networks, it can become production ready status. Breaking changes should be avoided as much as possible. All breaking changes require approval and announcement.
• Maintenance
  • Code owner should be responsible for bug fixes, PR review, discussion, and announcements.